Post Syndicated from Pat Patterson original https://www.backblaze.com/blog/use-a-cloudflare-worker-to-send-notifications-on-backblaze-b2-events/

When building an application or solution on Backblaze B2 Cloud Storage, a common requirement is to be able to send a notification of an event (e.g., a user uploading a file) so that an application can take some action (e.g., processing the file). In this blog post, I’ll explain how you can use a Cloudflare Worker to send event notifications to a wide range of recipients, allowing great flexibility when building integrations with Backblaze B2.

Why Use a Proxy to Send Event Notifications?

Event notifications are useful whenever you need to ensure that a given event triggers a particular action. For example, last month, I explained how a video sharing site running on Vultr’s Infrastructure Cloud could store raw and transcoded videos in Backblaze B2. In that example, when a user uploaded a video to a Backblaze B2 bucket via the web application, the web app sent a notification to a Worker app instructing the Worker to read the raw video file from the bucket, transcode it, and upload the processed file back to Backblaze B2.

A drawback of this approach is that, if we were to create a mobile app to upload videos, we would have to copy the notification logic into the mobile app. As the system grows, so does the maintenance burden. Each new app needs code to send notifications and, worse, if we need to add a new field to the notification message, we have to update all of the apps. If, instead, we move the notification logic from the web application to a Cloudflare Worker, we can send notifications on Backblaze B2 events from a single location, regardless of the origin of the request. This pattern of wrapping an API with a component that presents the exact same API but adds its own functionality is known as a proxy.

Cloudflare Workers: A Brief Introduction

Cloudflare Workers provides a serverless execution environment that allows you to create applications that run on Cloudflare’s global edge network. A Cloudflare Worker application intercepts all HTTP requests destined for a given domain, and can return any valid HTTP response. Your Worker can create that HTTP response in any way you choose. Workers can consume a range of APIs, allowing them to directly interact with the Cloudflare cache, manipulate globally unique Durable Objects, perform cryptographic operations, and more.

Cloudflare Workers often, but not always, implement the proxy pattern, sending outgoing HTTP requests to servers on the public internet in the course of servicing incoming requests. If we implement a proxy that intercepts requests from clients to Backblaze B2, it could both forward those requests to Backblaze B2 and send notifications of those requests to one or more recipient applications.

This example focuses on proxying requests to the Backblaze S3 Compatible API, and can be used with any S3 client application that works with Backblaze B2 by simply changing the client’s endpoint configuration.

Implementing a similar proxy for the B2 Native API is much simpler, since B2 Native API requests are secured by a bearer token rather than a signature. A B2 Native API proxy would simply copy the incoming request, including the bearer token, changing only the target URL. Look out for a future blog post featuring a B2 Native API proxy.

Proxying Backblaze B2 Operations With a Cloudflare Worker

S3 clients send HTTP requests to the Backblaze S3 Compatible API over a TLS-secured connection. Each request includes the client’s Backblaze Application Key ID (access key ID in AWS parlance) and is signed with its Application Key (secret access key), allowing Backblaze B2 to authenticate the client and verify the integrity of the request. The signature algorithm, AWS Signature Version 4 (SigV4), includes the Host header in the signed data, ensuring that a request intended for one recipient cannot be redirected to another. Unfortunately, this is exactly what we want to happen in this use case!

Our proxy Worker must therefore validate the signature on the incoming request from the client, and then create a new signature that it can include in the outgoing request to the Backblaze B2 endpoint. Note that the Worker must be configured with the same Application Key and ID as the client to be able to validate and create signatures on the client’s behalf.

Here’s the message flow:

1. A user performs an action in a Backblaze B2 client application, for example, uploading an image.
2. The client app creates a signed request, exactly as it would for Backblaze B2, but sends it to the Cloudflare Worker rather than directly to Backblaze B2.
3. The Worker validates the client’s signature, and creates its own signed request.
4. The Worker sends the signed request to Backblaze B2.
5. Backblaze B2 validates the signature, and processes the request.
6. Backblaze B2 returns the response to the Worker.
7. The Worker forwards the response to the client app.
8. The Worker sends a notification to the webhook recipient.
9. The recipient takes some action based on the notification.

These steps are illustrated in the diagram below.

The validation and signing process imposes minimal overhead, even for requests with large payloads, since the signed data includes a SHA-256 digest of the request payload, included with the request in the x-amz-content-sha256 HTTP header, rather than the payload itself. The Worker need not even read the incoming request payload into memory, instead passing it to the Cloudflare Fetch API to be streamed directly to the Backblaze B2 endpoint.

The Worker returns Backblaze B2’s response to the client unchanged, and creates a JSON-formatted webhook notification containing the following parameters:

• contentLength: Size of the request body, if there was one, in bytes.
• contentType: Describes the request body, if there was one. For example, image/jpeg.
• method: HTTP method, for example, PUT.
• signatureTimestamp: Request timestamp included in the signature.
• status: HTTP status code returned from B2 Cloud Storage, for example 200 for a successful request or 404 for file not found.
• url: The URL requested from B2 Cloud Storage, for example, https://s3.us-west-004.backblazeb2.com/my-bucket/hello.txt.

The Worker submits the notification to Cloudflare for asynchronous processing, so that the response to the client is not delayed. Once the interaction with the client is complete, Cloudflare POSTs the notification to the webhook recipient.

Prerequisites

If you’d like to follow the steps below to experiment with the proxy yourself, you will need to:

• Sign up for a Backblaze B2 account. You’ll receive 10GB of storage, free of charge, no credit card required.
• Sign up for a CloudflareWorkers account. You’ll be able to publish Workers to the default *.workers.dev subdomain free of charge, or to your own paid domain.
• Install and configure the Workers CLI, wrangler.

1. Creating a Cloudflare Worker Based on the Proxy Code

The Cloudflare Worker B2 Webhook GitHub repository contains full source code and configuration details. You can use the repository as a template for your own Worker using Cloudflare’s wrangler CLI. You can change the Worker name (my-proxy in the sample code below) as you see fit:

wrangler generate my-proxy
https://github.com/backblaze-b2-samples/cloudflare-b2-proxy
cd my-proxy

2. Configuring and Deploying the Cloudflare Worker

You must configure AWS_ACCESS_KEY_ID and AWS_S3_ENDPOINT in wrangler.toml before you can deploy the Worker. Configuring WEBHOOK_URL is optional—you can set it to empty quotes if you just want a vanity URL for Backblaze B2.

[vars]

AWS_ACCESS_KEY_ID = "<your b2 application key id>"
AWS_S3_ENDPOINT = "</your><your endpoint - e.g. s3.us-west-001.backblazeb2.com>"
AWS_SECRET_ACCESS_KEY = "Remove this line after you make AWS_SECRET_ACCESS_KEY a secret in the UI!"
WEBHOOK_URL = "<e.g. https://api.example.com/webhook/1 >"

Note the placeholder for AWS_SECRET_ACCESS_KEY in wrangler.toml. All variables used in the Worker must be set before the Worker can be published, but you should not save your Backblaze B2 application key to the file (see the note below). We work around these constraints by initializing AWS_SECRET_ACCESS_KEY with a placeholder value.

Use the CLI to publish the Worker project to the Cloudflare Workers environment:

wrangler publish

Now log in to the Cloudflare dashboard, navigate to your new Worker, and click the Settings tab, Variables, then Edit Variables. Remove the placeholder text, and paste your Backblaze B2 Application Key as the value for AWS_SECRET_ACCESS_KEY. Click the Encrypt button, then Save. The environment variables should look similar to this:

Finally, you must remove the placeholder line from wrangler.toml. If you do not do so, then the next time you publish the Worker, the placeholder value will overwrite your Application Key.

You can access the Worker via its default endpoint, which will have the form https://my-proxy.<your-workers-subdomain>.workers.dev, or create a DNS record in your own domain and configure a route associating the custom URL with the Worker.

If you try accessing the Worker URL via the browser, you’ll see an error message:

<Error>
<Code>AccessDenied</Code>
<Message>
Unauthenticated requests are not allowed for this api
</Message>
</Error>

This is expected—the Worker received the request, but the request did not contain a signature.

3. Configuring the Client Application

The only change required in your client application is the S3 endpoint configuration. Set it to your Cloudflare Worker’s endpoint rather than your Backblaze account’s S3 endpoint. As mentioned above, the client continues to use the same Application Key and ID as it did when directly accessing the Backblaze S3 Compatible API.

4. Implementing a Webhook Consumer

The webhook consumer must accept JSON-formatted messages via HTTP POSTs at a public endpoint accessible from the Cloudflare Workers environment. The webhook notification looks like this:

{
"contentLength": 30155,
"contentType": "image/png",
"method": "PUT",
"signatureTimestamp": "20220224T193204Z",
"status": 200,
"url": "https://s3.us-west-001.backblazeb2.com/my-bucket/image001.png"
}

You might implement the webhook consumer in your own application or, alternatively, use an integration platform such as IFTTT, Zapier, or Pipedream to trigger actions in downstream systems. I used Pipedream to create a workflow that logs each Backblaze B2 event as a new row in a Google Sheet. Watch it in action in this short video:

Put the Proxy to Work!

The Cloudflare Worker/Backblaze B2 Proxy can be used as-is in a wide variety of integrations—anywhere you need an event in Backblaze B2 to trigger an action elsewhere. At the same time, it can be readily adapted for different requirements. Here are a few ideas.

In this initial implementation, the client uses the same credentials to access the Worker as the Worker uses to access Backblaze B2. It would be straightforward to use different credentials for the upstream and downstream connections, ensuring that clients can’t bypass the Worker and access Backblaze B2 directly.

POSTing JSON data to a webhook endpoint is just one of many possibilities for sending notifications. You can integrate the worker with any system accessible from the Cloudflare Workers environment via HTTP. For example, you could use a stream-processing platform such as Apache Kafka to publish messages reliably to any number of consumers, or, similarly, send a message to an Amazon Simple Notification Service (SNS) topic for distribution to SNS subscribers.

As a final example, the proxy has full access to the request and response payloads. Rather than sending a notification to a separate system, the worker can operate directly on the data, for example, transparently compressing incoming uploads and decompressing downloads. The possibilities are endless.

How will you put the Cloudflare Worker Backblaze B2 Proxy to work? Sign up for a Backblaze B2 account and get started!

The post Use a Cloudflare Worker to Send Notifications on Backblaze B2 Events appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Jennifer Newman original https://www.backblaze.com/blog/a-refreshing-partnership-backblaze-and-soda/

Moving all your stuff is one of the most paralyzing projects imaginable. Which is why professional movers are amazing: one tackles the dishes, a couple folks hit the mattresses and closets. And there’s one guy (probably the rookie) who gets assigned to the junk drawers and odd gadgets. Suddenly, your old house is empty and your life is safe and orderly in boxes moving across the country.

Now imagine moving your businesses’ most valuable data across the country or the world—whether it’s the organization, the security, the budgeting, or all of the above and then some—it can be absolutely paralyzing, even when your current data storage approach is holding you back and you know you need to make a change.

This is where SoDA comes in.

Essentially your professional movers in the cloud, the SoDA team analyzes your cloud or on-prem infrastructure and then orchestrates the movement, replication, or syncing of data to wherever you want it to go—limiting any downtime in the process and ensuring your data is secure in flight and structured exactly as you need it in its new home. If deciding where to send data is an issue, they’ll use the analysis of your existing setup to scope the best solution by value for your business.

The Backblaze and SoDA Partnership leverages SoDA’s data movement services to unlock Backblaze B2 Cloud Storage’s value for more businesses. The partnership offers the following benefits:

• A cost analysis of your existing storage infrastructure.
• A “dry run” feature that compares existing storage costs to new storage costs and any transfer costs so you “know before you go.”
• The ability to define policies for how the data should move and where.
• Flexibility to move, copy, sync, or archive data to Backblaze B2.
• Migration and management via the Backblaze S3 Compatible API—easily migrate data, and then develop and manage both on-prem and cloud data via the API going forward.

Why Should You Try Backblaze and SoDA?

First: Backblaze will pay for SoDA’s services for any customer who agrees to migrate 10TB or more and commit to maintaining at least 10TB in Backblaze B2 for a minimum of one year.*

People don’t believe this when we tell them, but we’ll say it again: You won’t receive an invoice for your initial data migration, ever.

If that’s not reason enough to run a proof of concept, here’s more to think about:

Moving a couple of files to the cloud is easy peasy. But what happens if you have billions of files structured in multiple folders across multiple storage locations? You could use legacy tools or command line tools, but all of the scripting and resource management for the data in flight will be on you. You’re smart enough to do it, but if someone else is willing to pay the metaphorical movers, why deal with the hassle?

With SoDA, you do not have to worry about any of it. You define your source locations, define your destination Backblaze B2 bucket and start a transfer. SoDA takes care of the rest. That is truly easy peasy.

An Example of the Backlaze and SoDA Value Proposition

One customer we recently worked with was managing data in their own data center and having issues with reliability and SLAs for current customers. They needed availability at 99.999% as well as cost-effectiveness for future scaling. They identified Backblaze B2 as a provider that checked both boxes and Backblaze recommended SoDA for the move. The customer migrated 1PB of data (over a billion files) into B2 Cloud Storage. Other than making the decision and pointing where the data should go, the customer didn’t have to lift a finger.

Try It Today

If you’re not convinced yet, the SoDA and Backblaze teams are ready to make your life easier at any time. You can schedule a meeting here. Or you can check out the Quickstart guide to explore the solution today.

*Conditions may apply.

The post A Refreshing Partnership: Backblaze and SoDA appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Molly Clancy original https://www.backblaze.com/blog/ransomware-takeaways-from-q1-2022/

The impact of the war in Ukraine is evolving in real time, particularly when it comes to the ransomware landscape. Needless to say, it dominated the ransomware conversation throughout Q1 2022. This quarter, we’re digging into some of the consequences from the invasion and what it means for you in addition to a few broader domestic developments.

Why? Staying up to date on ransomware trends can help you prepare your security infrastructure in the short and long term to protect your valuable data. In this series, we share five key takeaways based on what we saw over the previous quarter. Here’s what we observed in Q1 2022.

1. Sanctions and International Attention May Have Depressed Some Ransomware Activity

Following the ground invasion, ransomware attacks seemed to go eerily quiet especially when government officials predicted cyberattacks could be a key tactic. That’s not to say attacks weren’t being carried out without being reported, but the radio silence was notable enough that a few media outlets wondered why.

International attention may be one reason—cybercriminals tend to be wary of the spotlight. Having the world’s eyes on a region where much cybercrime originates seems to have pushed cybercriminals into the shadows. The sanctions imposed on Russia have made it more difficult for cybercrime syndicates based in the country to receive, convert, and disperse payment from victims. The war also may have caused some chaos within ransomware syndicates and fomented fears that cyberinsurers would not pay for claims. As a result, we’ve seen a slowing of ransomware incidents in the first quarter, but that may not last.

Key Takeaway: While ransomware attacks may be down short-term, no one should be lulled into thinking the threat is gone, especially with government agencies on high alert and warnings from the highest levels that businesses should still be on guard.

2. Long-term Socioeconomic Impacts Could Trigger a New Wave of Cybercrime

As part of their ongoing analysis, cyber security consultants Coveware, illustrated how the socioeconomic precarity caused by sanctions could lead to a larger number of people turning to cybercrime as a way to support themselves. In their reporting, they analyzed the number of trained cyber security professionals who they’d expect to be out of work given Russia’s rising unemployment rate in order to estimate a pool of potential new ransomware operators. To double the number of individuals currently acting as ransomware operators, they found that only 7% of the newly unemployed workforce would have to convert to cybercrime.

They note, however, that it remains to be seen what impact a larger labor pool would have since new entrants looking for fast cash may not be as willing to put in the time and effort to carry out big game tactics that typified the first half of 2021. As such, Coveware would expect to see an increase in attacks on small to medium-sized enterprises (which already make up the largest portion of ransomware victims today) and a decline in ransom demands with new operators hoping to make paying up more attractive for victims.

Key Takeaway: If the threat materializes, new entrants to the ransomware game are likely to try to fly under the radar, which means we would expect to see a larger number of small to medium-sized businesses targeted with ransoms that won’t make headlines, but that nonetheless hurt the businesses affected.

3. One Ransomware Operator Paid the Price for Russian Allegiance; Others Declared Neutrality

In February, ransomware group Conti declared their support for Russian actions and threatened to retaliate against Western entities targeting Russian infrastructure. But Conti appears to have miscalculated the loyalty of its affiliates, many of whom are likely pro-Ukraine. The declaration backfired when one of their affiliates leaked chat logs following the announcement. Shortly after, LockBit, another prolific ransomware group, took a cue from Conti’s blunder, declaring neutrality and swearing off any attacks against Russia’s many enemies. Their reasoning? Surprisingly inclusive for an organized crime syndicate:

“Our community consists of many nationalities of the world, most of our pentesters are from the CIS including Russians and Ukrainians, but we also have Americans, Englishmen, Chinese, French, Arabs, Jews, and many others in our team… We are all simple and peaceful people, we are all Earthlings.”

As we know, the ransomware economy is a wide, interconnected network of actors with varying political allegiances. The actions of LockBit may assuage some fears that Russia would be able to weaponize the cybercrime groups that have been allowed to operate with impunity within its borders, but that’s no reason to rest easy.

Key Takeaway: LockBit’s actions and words reinforce the one thing we know for sure about cybercriminals: Despite varying political allegiances, they’re unified by money and they will come after it if it’s easy for the taking.

4. CISA Reports the Globalized Threat of Ransomware Increased in 2021

The U.S. Cybersecurity and Infrastructure Security Agency (CISA) released a statement in March summarizing the trends they saw throughout 2021. They outlined a number of tactics that we saw throughout the year as well, including:

• Targeting attacks on holidays and weekends.
• Targeting managed service providers.
• Targeting backups stored in on-premises devices and in the cloud.

Among others, these tactics pose a threat to critical infrastructure, healthcare, financial institutions, education, businesses, and nonprofits globally.

Key Takeaway: The advisory outlines 18 mitigation strategies businesses and organizations can take to protect themselves from ransomware, including some of the top strategies as we see it: protecting cloud storage by backing up to multiple locations, requiring MFA for access, and encrypting data in the cloud.

5. Russia Could Use Ransomware to Offset Sanctions

Despite our first observation that ransomware attacks slowed somewhat early in the quarter, the Financial Crimes Enforcement Network (FinCEN) issued an alert in March that Russia may employ state-sponsored actors to evade sanctions and bring in cryptocurrency by ramping up attacks. They warned financial institutions, specifically, to be vigilant against these threats to help thwart attempts by state-sponsored Russian actors to extort ransomware payments.

The warnings follow an increase in phishing and distributed denial-of-service (DDoS) attacks that have persisted throughout the year and increased toward the end of February into March as reported by Google’s Threat Analysis Group. In reports from ThreatPost covering the alert as well as Google’s observations, cybersecurity experts seemed doubtful that ransomware payouts would make much of a dent in alleviating the sanctions, and noted that opportunities to use ransomware were more likely on an individual level.

Key Takeaway: The warnings serve as a reminder that both individual actors and state-sponsored entities have ransomware tools at their disposal to use as a means to retaliate against sanctions or simply support themselves, and that the best course of action is to shore up defenses before the anticipated threats materialize.

What This All Means for You

The changing political landscape will continue to shape the ransomware economy in new and unexpected ways. Being better prepared to avoid or mitigate the effects of ransomware makes more and more sense when you can’t be sure what to expect. Ransomware protection doesn’t have to be costly or confusing. Check out our ransomware protection solutions to get started.

The post Ransomware Takeaways From Q1 2022 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/announcing-backblaze-b2-universal-data-migration/

Your data is valuable. Whether you’re sequencing genomes, managing a media powerhouse, or running your own business, you need fast, affordable, ready access to it in order to achieve your goals. But, you can’t get the most out of your data if it’s locked-in to a provider where it’s hard to manage or time-consuming to retrieve. Unfortunately, due to egress fees and closed, “all-in-one” platforms, vendor lock-in is currently trapping too many companies.

Backblaze can help: Universal Data Migration, a new service launched today, covers all data transfer costs, including legacy provider egress fees, and manages data migration from any legacy on-premises or cloud source. In short, your migration to Backblaze B2 Cloud Storage is on us.

Many of the businesses we’ve spoken to about this didn’t believe that the service was free at first. But seriously—you will never see an invoice for your transfer fees, any egress fees levied by your legacy vendor when you pull data out, or for the assistance in moving data.

If you’re still in doubt, read on to learn more about how Universal Data Migration can help you say goodbye to vendor lock-in, cold delays, and escalating storage costs, and hello to B2 Cloud Storage gains, all without fears of cost, complexity, downtime, or data loss.

How Does Universal Data Migration Work?

Backblaze has curated a set of integrated services to handle migrations from pretty much every source, including:

• Public cloud storage
• Servers
• Network attached storage (NAS)
• Storage area networks (SAN)
• Tape/LTO solutions
• Cloud drives

We cover data transfer and egress costs and facilitate the migration to Backblaze B2 Cloud Storage. The turnkey service expands on our earlier Cloud to Cloud Migration services as well as transfers via internet and the Backblaze Fireball rapid ingest devices. These offerings are now rolled up into one universal service.

“I thought moving my files would be the hardest part of the process, and it’s why I never really thought about switching providers before, but it was easy.”
—Tristan Pelligrino, Co-founder, Motion

We do ask that companies who use the service commit to maintaining at least 10TB in Backblaze B2 for a minimum of one year, but we expect that our cloud storage pricing—a quarter the cost of comparable services—and our interoperability with other cloud services, will keep new customers happy for that first year and beyond.

Outside of specifics that will vary by your unique infrastructure and workflows, migration types include:

• Cloud to cloud: Reads from public cloud storage or a cloud drive (e.g., Amazon S3 or Google Drive) and writes to Backblaze B2 via inter-cloud bandwidth.
• On-premises to cloud: Reads from a server, NAS, or SAN and writes to Backblaze B2 over optimized cloud pipes or via Backblaze’s 96TB Fireball rapid ingest device.
• LTO/tape to cloud: Reads tape media, from reel cassettes to cartridges and more, and writes to Backblaze B2 via a high-speed, direct connection.

Backblaze also supports simple internet transfers for moving files over your existing bandwidth—with multi-threading to maximize speed.

How Much Does Universal Data Migration Cost?

Not to sound like a broken record, but this is the best part—the service is entirely free to you. You’ll never receive a bill. Backblaze incurs all data transfer and legacy vendor egress or download fees for inbound migrations >10TB with a one-year commitment. It’s pretty cool that we can help save you money; it’s even cooler that we can help more businesses build the tech stacks they want using unconflicted providers to truly get the most out of their data.

“In the cloud space, the biggest complaint that we hear from clients is the cost of egress and storage. With Backblaze, we saved money on the migration, but also overall on the storage and the potential future egress of this data.”
—Tom Kehn, Senior Solutions Architect at CHESA, Fortune Media’s technology systems integrator

Even More Benefits

What else do you get with Universal Data Migration? Additional benefits include:

• Truly universal migrations: Secure data mobility from practically any source.
• Support along the way: Simple, turnkey services with solution engineer support to help ensure easy success.
• Safe and speedy transfers: Proven to securely transfer millions of objects and petabytes of data, often in just days.

Ready to Get Started?

The Universal Data Migration service is generally available now. To qualify, organizations must migrate and commit to maintaining at least 10TB in Backblaze B2 for a minimum of one year. For more information or to set up a free proof of concept, contact the Backblaze Sales team.

The post Announcing Backblaze B2’s Universal Data Migration appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Elton Carneiro original https://www.backblaze.com/blog/backblaze-partner-api-the-details/

Last week, we announced enhancements to our partner program that make working with Backblaze even easier for current and future partners. We shared a deep dive into the first new offering—Backblaze B2 Reserve—on Friday, and today, we’re digging into another key element: the Backblaze Partner API. The Backblaze Partner API enables independent software vendors (ISVs) participating in Backblaze’s Alliance Partner program to add Backblaze B2 Cloud Storage as a seamless backend extension within their own platform.

Read on to learn more about the Backblaze Partner API and what it means for existing and potential Alliance Partners.

What Is the Backblaze Partner API?

With the Backblaze Partner API, ISVs participating in Backblaze’s Alliance Partner program can programmatically provision accounts, run reports, and create a bundled solution or managed service which employs B2 Cloud Storage on the back end while delivering a unified experience to their users.

By unlocking an improved customer experience, the Partner API allows Alliance Partners to build additional cloud services into their product portfolio to generate new revenue streams and/or grow existing margin.

Why Develop the Backblaze Partner API?

We heard frequently from our existing partners that they wanted to provide a more seamless experience for their customers when it came to offering a cloud storage tier. Specifically, they wanted to keep customers on their site rather than requiring them to go elsewhere as part of the sign up experience. We built the Partner API to deliver this enhanced customer experience while also helping our partners extend their services and expand their offerings.

“Our customers produce thousands of hours of content daily, and, with the shift to leveraging cloud services like ours, they need a place to store both their original and transcoded files. The Backblaze Partner API allows us to expand our cloud services and eliminate complexity for our customers—giving them time to focus on their business needs, while we focus on innovations that drive more value.”
—Murad Mordukhay, CEO, Qencode

What Does the Partner API Do, Specifically?

To create the Backblaze Partner API, we exposed existing functionality to allow partners to automate tasks like creating and ejecting member accounts, managing Groups, and leveraging system-generated reports to get granular billing and usage information—outlining user tasks individually so users can be billed more accurately for what they’ve used.

The API calls are:

• Account creation (adding Group members).
• Organizing accounts in Groups.
• Listing Groups.
• Listing Group members.
• Ejecting Group members.

Once the Partner API is configured, developers can use the Backblaze S3 Compatible API or the Backblaze B2 Native API to manage Group members’ Backblaze B2 accounts, including: uploading, downloading, and deleting files, as well as creating and managing the buckets that hold files.

How to Get Started With the Backblaze Partner API

If you’re familiar with Backblaze, getting started is straightforward:

1. Create a Backblaze account.
2. Enable Business Groups and B2 Cloud Storage.
3. Contact Sales for access to the API.
4. Create a Group.
5. Create an Application Key and set up Partner API calls.

Check out our documentation for more detailed information on getting started with the Backblaze Partner API. You can also reach out to us via email at any time to schedule a meeting to discuss how the Backblaze Partner API can help you create an easier customer experience.

The post Backblaze Partner API: The Details appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Elton Carneiro original https://www.backblaze.com/blog/backblaze-b2-reserve-the-details/

Yesterday, we announced enhancements to our partner program that make working with Backblaze even easier for current and prospective partners. Today, we’re digging into a key offering from that program: Backblaze B2 Reserve. Backblaze B2 Reserve brings more value to the Backblaze community of Channel Partners and opens up our easy, affordable cloud storage to many more.

Read on to learn more about Backblaze B2 Reserve and what it means for existing and potential Channel Partners.

What Is Backblaze B2 Reserve?

Predictable, affordable pricing is our calling card, but for a long time our Channel Partners have had a harder time than other customers when it came to accessing this value. Backblaze B2 Reserve brings them a capacity-based, annualized SKU which works seamlessly with channel billing models. The offering also provides seller incentives, Tera-grade support, and expanded migration services to empower the channel’s acceleration of cloud storage adoption and revenue growth.

Why Launch Backblaze B2 Reserve?

Short story, we heard a lot of feedback from our partners about how much they loved working with Backblaze B2 Cloud Storage, except for the service’s pricing model—which limited their ability to promote it to customers. Backblaze B2 is charged on a consumption-based model, meaning you only pay for what you use. This works great for many of our customers who value pay-as-you-go pricing, but not as well for those who value fixed, predictable, monthly or annual bills.

Customers who are more accustomed to planning for storage provisioning want to pay for cloud storage on a capacity-based model similar to how they would for on-premises storage. They buy what they expect to use up front, and their systems and processes are set up to utilize storage in that way. Additionally, the partners who include Backblaze B2 as part of packages they sell to their customers wanted predictable pricing to make things easier in their sales processes.

Backblaze B2 Reserve is a pricing package built to answer these needs—serving the distributors and value-added resellers who want to be able to present B2 Cloud Storage to their current and prospective customers.

How Does Backblaze B2 Reserve Work?

The Backblaze B2 Reserve offering is capacity-based, starting at 20TB, with key features, including:

• Free egress up to the amount of storage purchased per month.
• Free transaction calls.
• Enhanced migration services.
• No delete penalties.
• Tera support.

A customer can purchase more storage by buying 10TB add ons. If you’re interested in participating or just want to learn more, you can reach out to us via email to schedule a meeting.

How Is Backblaze B2 Reserve Different From Backblaze B2?

The main difference between Backblaze B2 Reserve and Backblaze B2 is the way the service is packaged and sold. Backblaze B2 uses a consumption model—you pay for what you use. Backblaze B2 Reserve uses a capacity model—you pay for a specific amount of storage up front.

“Backblaze’s ease and reliability, paired with their price leadership, has always been attractive, but having their pricing aligned with our business model will bring them into so many more conversations we’re having across the types of customers we work with.”
—Mike Winkelmann, Cinesys-Oceana

Ready to Get Started?

If you’re going to NAB, April 23-27th, we’ll be there, and we’d love to see you—click here to book a meeting. We’ll also be at the Channel Partners Conference, April 11-14th. Otherwise, reach out to us via email to schedule a chat. Let’s talk about how the new program can move your business forward.

The post Backblaze B2 Reserve: The Details appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Elton Carneiro original https://www.backblaze.com/blog/announcing-partner-program-enhancements/

Here at Backblaze, we can definitively say that we get by with a little (okay, a lot of) help from our friends. We’ve always been committed to building an open, transparent, and interoperable ecosystem which has helped us grow an incredible partner network. We provide easy, affordable, and trusted cloud storage as a neutral partner, and they provide all manner of services, products, and solutions that use our storage. But there’s always room for improvement, right?

Which is why, today, we’re enhancing our partner program with two major new offerings:

• Backblaze B2 Reserve: A predictable, capacity pricing model to empower our Channel Partners.
• Backblaze Partner API: A new API that empowers our Alliance Partners to easily integrate and manage B2 Cloud Storage within their products and platforms.

Read on to learn a bit more about each component, and stay tuned throughout this week and next for deeper dives into each element.

Capacity Pricing With Backblaze B2 Reserve

Backblaze B2 Reserve is a new offering for our Channel Partners. Predictable, affordable pricing is our calling card, but for a long time our Channel Partners had a harder time than other customers when it came to accessing this value. Backblaze B2 Reserve brings them a capacity-based, annualized SKU which works seamlessly with channel billing models. The offering also provides seller incentives, Tera-grade support, and expanded migration services to empower the channel’s acceleration of cloud storage adoption and revenue growth.

The key benefits include:

• Enhanced margin opportunity and a predictable pricing model.
• Easier conversations with customers accustomed to an on-premises or capacity model.
• Discounts and seller incentives.

The program is capacity based, starting at 20TB, with key features, including:

• Free egress up to the amount of storage purchased per month.
• Free transaction calls.
• Enhanced migration services.
• No delete penalties.
• Tera support.

It’s all of the same great functionality folded in. Partners get more margin, seller incentives, and a predictable growth model for customers.

“Backblaze’s ease and reliability, paired with their price leadership, has always been attractive, but having their pricing aligned with our business model will bring them into so many more conversations we’re having across the types of customers we work with.”
—Mike Winkelmann, Owner of CineSys Inc.

User Management and Usage Reporting With the Backblaze Partner API

The Backblaze Partner API empowers independent software vendors participating in Backblaze’s Alliance Partner Program to add Backblaze B2 Cloud Storage as a seamless backend extension within their own platform, where they can programmatically provision accounts, run reports, and create a bundled solution or managed service for a unified user experience. By unlocking an improved customer experience for the partner, the Partner API allows Alliance Partners to build additional cloud services into their product portfolio to generate new revenue streams and/or grow existing margin.

Features of the Partner API include:

• Account provisioning.
• Managing a practically unlimited number of accounts or groups.
• Comprehensive usage reporting.

In using the Partner API, partners can offer a proprietary branded, bundled solution with a unified bill, or create a solution that is “Powered by Backblaze B2.”

“Our customers produce thousands of hours of content daily, and, with the shift to leveraging cloud services like ours, they need a place to store both their original and transcoded files. The Backblaze Partner API allows us to expand our cloud services and eliminate complexity for our customers—giving them time to focus on their business needs, while we focus on innovations that drive more value.”
—Murad Mordukhay, CEO at Qencode

Other Benefits

To unlock the value inherent in Backblaze B2 Reserve and the Partner API, Backblaze is offering free migration to help customers painlessly copy or migrate their data from practically any source into B2 Cloud Storage.

This service supports truly free data mobility without complexity or downtime, including coverage of all data transfer costs and any egress fees charged by legacy vendors. Stay tuned for more on this feature that benefits both partners and all of our customers.

The addition of Tera support brings the benefit of a four-hour target response time for email support and named customer contacts to ensure that partners and their end users can troubleshoot at speed.

What’s Next?

These are the first of many features and programs that Backblaze will be rolling out this year to make our partners’ experience working with us better. Tomorrow, we’ll dive deeper into the Backblaze B2 Reserve offering. On Monday, we’ll offer more detail on the Backblaze Partner API feature. In the coming months, we’ll be sharing even more. Stay tuned.

Want to Learn More?

Reach out to us via email to schedule a meeting. If you’re going to the Channel Partners Conference, April 11–14th, we’ll be there and we’d love to see you! If not, reach out and we’d be happy to start a conversation about how the new program can move your business forward.

The post Announcing Partner Program Enhancements appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Jennifer Newman original https://www.backblaze.com/blog/bring-your-file-system-to-the-cloud-with-backblaze-ctera/

You know your file system. You love your file system (suspend your disbelief for a moment). You can find what you need (mostly). The hierarchy makes sense (kind of). Anyway—the point is, it’s convenient and it works.

You also know object storage offers scalability and cost savings that you’d love to tap into to either replace or extend your on-premises file servers or NAS. But how do you move your data to the cloud safely and securely without overhauling your entire file system to work with object storage?

Through a new partnership with CTERA, you can extend your corporate file system to Backblaze B2 Cloud Storage while maintaining all of your existing permissions, file structures, and security protocols.

The joint solution unlocks meaningful opportunities for small and medium-sized enterprises. Through the partnership, you can:

• Store all of your unstructured data in one centralized place, while maintaining instant and reliable access.
• Retire legacy file servers and NAS devices along with the time and expense required to maintain them.
• Empower a remote workforce and multi-site collaboration.
• Establish a resilient disaster recovery, business continuity, and ransomware mitigation plan.
• Optimize your budget with pay-as-you-go cloud storage pricing that’s a quarter the price of equivalent offerings.

“If you’re tired of buying new equipment every three years; replacing hard drives; paying for maintenance, or power, or space in a data center; and all of the headaches of managing remote user access, then the CTERA and Backblaze partnership is perfect for you. The setup is incredibly easy and the immediate budget and staffing relief will give you resources to tackle new opportunities. You’ll never have to—or want to—upgrade your NAS again.”
—Nilay Patel, VP of Sales, Backblaze

How It Works

CTERA’s Enterprise File Services Platform, through their core global file system technology, extends the capabilities of traditional NAS and file servers to the cloud. The joint solution provides fast local file access via cached files in CTERA’s Edge Filers, while storing the primary copy of your data in hot storage with Backblaze B2 for just $5/TB per month with a 99.9% uptime SLA.

Users across your enterprise have access to shared files via their Windows, Mac, or mobile devices, and data generated at the edge is automatically backed up to Backblaze B2 and accessible across your organization. The flexibility and extensibility of the Backblaze B2 + CTERA partnership consolidates your remote IT infrastructure and reduces the burden on your IT team to manage remote devices.

For customers that have distributed sites across the U.S. and EU, regions can be defined within CTERA and Backblaze B2 to ensure data is GDPR compliant. CTERA also offers an SDK for developers, enabling them to automatically configure Edge Filers and synchronize to Backblaze B2 Cloud Storage using infrastructure as code.

“We’re seeing a massive shift from traditional NAS to cloud NAS, from edge to core access, as organizations evolve and expand. CTERA is committed to providing the widest choice of cloud to our customers. The Backblaze-CTERA partnership establishes a compelling, new, cost-effective storage option for companies that wish to tier their data to the cloud for redundancy and collaboration.”
—Oded Nagel, Chief Strategy Officer, CTERA

About CTERA

CTERA is the edge-to-cloud file services leader, powering more than 50,000 connected sites and millions of corporate users. CTERA offers the industry’s most feature-rich global file system, enabling enterprises to centralize file access from any edge location or device without compromising performance or security. The CTERA Enterprise File Services Platform makes it easy for organizations to consolidate legacy NAS, backup and disaster recovery systems, and collaboration platforms while reducing costs by up to 80% versus legacy solutions. CTERA is trusted by the world’s largest companies, including McDonald’s, GE, Unilever, and Live Nation, as well as the U.S. Department of Defense and other government organizations worldwide.

Interested in Learning More?

Join us for a webinar on April 27, 2022 at 8 a.m. PST/11 a.m. EST to discover how to tier your file structure intelligently into Backblaze B2 with CTERA—register here. Anyone interested in exploring the solution today can check out the CTERA Solution Brief.

The post Bring Your File System to the Cloud With Backblaze + CTERA appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Molly Clancy original https://www.backblaze.com/blog/how-to-scale-a-storage-heavy-startup/

No developer likes to feel stuck with a cloud vendor, especially when you’re paying for the trap. Being locked in because the expense of moving is prohibitive or the risk of service disruption is too great can be singularly frustrating. Unfortunately, Gavin Wade, Founder and CEO of CloudSpot, found himself in exactly that position.

Gavin shared how he restructured his cloud architecture (containing more than 700TB of images under management), enabling him to:

• Cut storage costs in half.
• Slow the rate at which storage costs compound by half.
• Cut data transfer costs by 90%.
• Lower breakeven customer volume.
• Increase margins.

Amazon Web Services gave CloudSpot free credits for storage. When those credits ran out, CloudSpot would have happily left to avoid escalating storage costs but felt trapped by high egress fees.

“We had a few internal conversations where we concluded that we were stuck with Amazon. That’s never a good feeling in business.”
—Gavin Wade, Founder & CEO, CloudSpot

How CloudSpot Solved for Escalating Storage Costs

In the short term, CloudSpot’s development team, led by their vice president of engineering, took a few key steps to manage costs:

1. They untangled their monolithic system into a cluster of microservices.
2. They moved to a Kubernetes environment where images upload directly to storage, then CloudSpot’s microservices retroactively query the data they need.

The transition to microservices made their infrastructure more nimble, but Gavin still had to reluctantly cut key promotional offers like free migration for prospective customers in order to maintain margins.

When Cost-cutting Measures Still Don’t Suffice

Even after optimizing workflows, storage costs continued to snowball. Namely:

• The service grew—customers were uploading five times the previous year’s volume.
• Gavin wanted to position the company for triple-digit growth in the upcoming year.

They decided to move their production data to Backblaze B2 Cloud Storage. The potential ROI of switching to Backblaze B2 was too substantial to ignore for a data-heavy startup, and Backblaze’s Cloud to Cloud Migration service allowed them to move 700TB of data in one day with zero transfer fees.

Migrating Storage Clouds Without Service Disruption

CloudSpot’s data is accessed frequently, and the CloudSpot development team had to make sure customers saw no disruptions. To do so, they supported both environments—on Amazon S3 and Backblaze B2—simultaneously for one week to ensure everything was working. Then, they disabled uploads to Amazon S3 and redirected new uploads to Backblaze B2.

“It was like changing the tires on a car while it’s flying down the road at 100 mph,” but a change that resulted in no loss of operational efficiency, speed, or reliability.
—Gavin Wade, Founder & CEO, CloudSpot

Cloud to Cloud Migration Is Not Out of Reach

Like many developers, Gavin thought he was trapped in a walled garden with Amazon S3. Improving CloudSpot’s workflows unlocked the switch to Backblaze B2, enabling CloudSpot to:

• Structure workflows using best-of-breed providers.
• Reintroduce free migration.
• Grow margins.
• Demonstrate savvy decision-making to future investors.

“Software margins are expected to be high. If you can take a big cut of that, it allows you to scale more rapidly. It just makes our story so much better, especially as a SaaS business looking to scale, grow, and raise capital.”
—Gavin Wade, Founder & CEO, CloudSpot

Unlocking Capacity to Scale With Backblaze B2

Read more about how CloudSpot overcame vendor lock-in to realize exponential growth, and check out our Cloud to Cloud Migration offer and partners—we’ll pay for your data transfer if you need to move more than 10TB out of Amazon.

The post How to Scale a Storage-heavy Startup appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Molly Clancy original https://www.backblaze.com/blog/how-multi-cloud-backups-outage-proof-data/

Bob Graw, Director of IT for CallTrackingMetrics, a leading marketing automation platform, put words to a concern that’s increasingly troubling for businesses of all sizes: “If the data center gets wiped out, what happens to our business?”

Lately, high profile cloud outages happen at a regular clip, and the only thing you can count on is that systems will fail at some point. Savvy IT leaders work from that assumption regardless of their primary cloud provider, because they know redundant fail-safes should always be part of your plan.

Bob shared how CallTrackingMetrics outage-proofed their data by establishing a vendor-independent, multi-cloud system. Read on to learn how they did it.

Multi-cloud Storage Protects Data From Disasters

As CallTrackingMetrics’ data grew over the years, so did their data backups. They stored more than a petabyte of backups with one vendor. It was a strategy they grew less comfortable with over time. Their concerns included:

• Operating with a single point of failure.
• Becoming locked in with one vendor.
• Maintaining compliance with data regulations.
• Diversifying their storage infrastructure within budget.

Multi-cloud Solves for a Single Point of Failure

Bob had thought about diversifying CallTrackingMetrics’ storage infrastructure for years, and as outages continued apace, he decided to pull the trigger. He sought out an additional storage vendor where CallTrackingMetrics could store a redundant copy of their backups for disaster recovery and business continuity. “You should always have at least two viable, ready to go backups. It costs real money, but if you can’t come back to life in a disaster scenario, it is basically game over,” Bob explained.

They planned to mirror data from their diversified cloud provider in Backblaze B2, creating a robust multi-cloud strategy. With data backups in two places, they would be better protected from outages and disasters.

“We trust the Backblaze technology. I’d be very surprised if I ever lost data with Backblaze.”
—Bob Graw, Senior Software Engineer, CallTrackingMetrics

Multi-cloud Solves for Vendor Lock-in

Diversifying storage providers came with the added benefit of solving for vendor lock-in. “We did not want to be stuck with one cloud provider forever,” Bob said. Addressing storage was only one part of that strategy, though. They also intentionally avoided using specific services from their diversified cloud vendor like elastic search and databases to keep their data portable. That way, “We could really take our system to anybody that provides compute or storage, and we would be fine,” Bob said.

Solving for Compliance

Some of CallTrackingMetrics’ clients work in highly regulated industries, so compliance with regulations like HIPAA and GDPR was important for them to maintain when searching for a new storage provider. Those regulations stipulate how data is stored, the security protocols in place to protect data, and data retention requirements. “We feel like Backblaze is very secure, and we rely on Backblaze being secure so that our backups are safe and we stay compliant,” Bob said.

Solving for Budget Concerns

Bob and CallTrackingMetrics Founder, Todd Fisher, had both used Backblaze Personal Backup for years, so that familiarity opened the door. But the Backblaze Cloud to Cloud Migration service sealed the deal. Due to high data transfer fees, “Getting out of our previous provider isn’t cheap,” Bob said. But with data transfer fees covered through the Cloud to Cloud Migration service, they addressed one of their primary concerns—staying within budget. On top of an easy migration, after making the switch, Bob saw an immediate 50% savings on his storage bill thanks to Backblaze’s affordable, single-tier pricing.

“With Backblaze, the benefits are threefold: We like the cost savings, we like that our eggs aren’t all in one basket, and Backblaze is super simple to use.”
—Bob Graw, Senior Software Engineer, CallTrackingMetrics

Cloud Storage Helps Leading Marketing Platform Level Up

Now that CallTrackingMetrics has a multi-cloud system to protect their data from outages, they can focus on solving for the next challenge—getting better visibility into their overall cloud usage. With the savings they recouped from moving backups to Backblaze B2, Bob was able to invest in Lacework, software that helps with automation to protect their multi-cloud environment.

Thinking about going multi-cloud but worried about the cost of transferring your data? Check out our Cloud to Cloud Migration service and get started today—the first 10GB are free.

“Backblaze is our ultimate security blanket. We know our big pile of data is safe and sound.”
—Bob Graw, Senior Software Engineer, CallTrackingMetrics

The post How Multi-cloud Backups Outage-proof Data appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Pat Patterson original https://www.backblaze.com/blog/media-transcoding-with-backblaze-b2-and-vultr-optimized-cloud-compute/

Since announcing the Backblaze + Vultr partnership last year, we’ve seen our mutual customers build a wide variety of applications combining Vultr’s Infrastructure Cloud with Backblaze B2 Cloud Storage, taking advantage of zero-cost data transfer between Vultr and Backblaze. This week, Vultr announced Optimized Cloud Compute instances, virtual machines pairing dedicated best-in-class AMD CPUs with just the right amount of RAM and NVMe SSDs.

To mark the occasion, I built a demonstration that both showcases this new capability and gives you an example application to adapt to your own use cases.

Imagine you’re creating the next big video sharing site—CatTube—a spin-off of Catblaze, your feline-friendly backup service. You’re planning all sorts of amazing features, but the core of the user experience is very familiar:

• A user uploads a video from their mobile or desktop device.
• The user’s video is available for viewing on a wide variety of devices, from anywhere in the world.

Let’s take a high-level look at how this might work…

Transcoding Explained: How Video Sharing Sites Make Videos Shareable

The user will upload their video to a web application from their browser or a mobile app. The web application must store the uploaded user videos in a highly scalable, highly available service—enter Backblaze B2 Cloud Storage. Our customers store, in the aggregate, petabytes of media data including video, audio, and still images.

But, those videos may be too large for efficient sharing and streaming. Today’s mobile devices can record video with stunning quality at 4K resolution, typically 3840 × 2160 pixels. While 4K video looks great, the issue is that even with compression, it’s a lot of data—about 1MB per second. Not all of your viewers will have that kind of bandwidth available, particularly if they’re on the move.

So, CatTube, in common with other popular video sharing sites, will need to convert raw uploaded video to one or more standard, lower-resolution formats, a process known as transcoding.

Transcoding is a very different workload from running a web application’s backend. Where an application server requires high I/O capability, but relatively little CPU power, transcoding is extremely CPU-intensive. You decide that you’ll need two sets of machines for CatTube—application servers and workers. The worker machines can be optimized for the transcoding task, taking advantage of the fastest available CPUs.

For these tasks, you need appropriate cloud compute instances. I’ll walk you through how I implemented CatTube as a very simple video sharing site with Backblaze B2 and Vultr’s Infrastructure Cloud using Vultr’s Cloud Compute instances for the application servers and their new Optimized Cloud Compute instances for the transcoding workers.

Building a Video Sharing Site With Backblaze B2 + Vultr

The video sharing example comprises a web application, written in Python using the Django web framework, and a worker application, also written in Python, but using the Flask framework.

Here’s how the pieces fit together:

1. The user uploads a video from their browser to the web app.
2. The web app uploads the raw video to a private bucket on Backblaze B2.
3. The web app sends a message to the worker instructing it to transcode the video.
4. The worker downloads the raw video to local storage and transcodes it, also creating a thumbnail image.
5. The worker uploads the transcoded video and thumbnail to Backblaze B2.
6. The worker sends a message to the web app with the addresses of the input and output files in Backblaze B2.
7. Viewers around the world can enjoy the video.

These steps are illustrated in the diagram below.

There’s a more detailed description in the Backblaze B2 Video Sharing Example GitHub repository, as well as all of the code for the web application and the worker. Feel free to fork the repository and use the code as a starting point for your own projects.

Here’s a short video of the system in action:

Want to Try It for Yourself?

Vultr’s new Cloud Compute Optimized instances are a perfect match for CPU-intensive tasks such as media transcoding. Zero-cost ingress and egress between Backblaze B2 and Vultr’s Infrastructure Cloud allow you to build high performance, scalable applications to satisfy a global audience. Sign up for Backblaze B2 and Vultr’s Infrastructure Cloud today, and get to work!

The post Media Transcoding With Backblaze B2 and Vultr Optimized Cloud Compute appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Molly Clancy original https://www.backblaze.com/blog/how-to-run-vfx-workflows-in-the-cloud/

An hour from Queens. An hour from Jersey. Two hours from Staten Island. That’s how long it would take Molecule VFX IT staff to travel from their homes to the closet in Manhattan that housed the team’s LTO device. All those hours, just to spend five minutes switching out one tape.

It was a huge waste of time, not to mention subway fares. The hassle of tape wasn’t the only reason Molecule decided to make their production workflows fully cloud-based, but the IT team certainly doesn’t mind skipping that trip these days.

Moving production entirely to the cloud allowed Molecule to unlock the value of their artists’ time as well as the IT staff to support them, and save money in the process. If your media team has been contemplating a fully cloud-based workflow, read on to learn how Molecule did it—including how they managed to maintain the ability to move data from the cloud back to tape on demand without maintaining on-premises tape infrastructure.

Molecule’s Case for the Cloud

Visual effects artists want to be able to hop into a new script, work on it, render it, review it, QC it, and call it done. Their work is the most valuable element of the business. Anything that gets in the way of that or slows down the workflow directly impacts the company’s success, and an on-premises system was doing exactly that.

• With IT staff working from home, LTO maintenance tied them up for hours—time that could have been spent helping Molecule’s visual effects artists create.
• Beyond tape, the team managed a whole system of machines, networks, and switches. Day-to-day issues could knock out the company’s ability to get work done for entire days.

They knew moving to the cloud would optimize staff time and mitigate those outages, but it didn’t happen overnight. Because much of their business already happens in the digital workspace, Molecule had been slowly moving to the cloud over the past few years. The shift to remote work due to the COVID-19 pandemic accelerated their transition.

Strategies for Moving VFX Workflows to the Cloud

Molecule’s Full Stack Software Architect, Ben Zenker, explained their approach. Through the process, he identified a few key strategies that made the transition a success, including:

• Taking a phased approach while deciding between hybrid and fully cloud-based workflows.
• Reading the fine print when comparing providers.
• Rolling their own solutions where possible.
• Thoroughly testing workflows.
• Repurposing on-premises infrastructure.

1. Take a Phased Approach

Early in the transition, the Molecule team was still using the tape system and an on-premises Isilon server for some workloads. Because they were still deciding if they were going to have a hybrid system or go fully cloud, they took an ad hoc approach to identifying what data was going to be in Backblaze B2 Cloud Storage and what production infrastructure was going to be in CoreWeave, a cloud compute partner that specializes in VFX workloads. Ben explained, “Once we decided definitively we wanted to be fully in the cloud, connecting CoreWeave and Backblaze was simple—if it was on CoreWeave, it was getting backed up in Backblaze B2 nightly.”

2. Read the Fine Print

The team planned to sync incremental backups to the cloud every night. That meant their data would change every day as staff deleted or updated files. They figured out early on that retention minimums were a non-starter. Some cloud providers charge for deleted data for 30, 60, or even 90 days, meaning Molecule would be forced to pay for storage on data they had deleted months ago. But not all cloud providers are transparent about their retention policies. Molecule took the time to track down these policies and compare costs.

“Backblaze was the only service that met our business requirements without a retention minimum.”
—Ben Zenker, Full Stack Software Architect, Molecule VFX

3. Roll Your Own Solutions Where Possible

The team creates a lot of their own web tools to interact with other technology, so it was a relatively easy lift to set up rclone commands to run syncs of their production data nightly to Backblaze B2. Using rclone, they also built a variable price reporting tool so that higher ups could easily price out different projects and catch potential problems like a runaway render.

“There are hundreds of options that you can pass into rclone, so configuring it involved some trial and error. Thankfully it’s open-source, and Backblaze has documentation. I made some small tweaks and additions to the tool myself to make it work better for us.”
—Ben Zenker, Full Stack Software Architect, Molecule VFX

4. Test and Test Again

In reflecting on the testing phase they went through, Ben acknowledges he could have been more liberal. He noted, “I went into it a little cautious because I didn’t want to end up incurring big charges for a test, but Backblaze has all sorts of safeguards in place. You can set price limits and caps, which was great for the testing period.”

5. Repurpose On-premises Infrastructure

The on-premises Isilon server and the physical tape system are no longer part of the active project workflow. They still utilized those devices to host some core services for a time—a firewall, authentication, and a VPN that some members used. In the end, they decided to fully retire all on-premises infrastructure, but repurposing the on-premises infrastructure allowed them to maximize its useful life.

But What If Clients Demand Tape?

While Molecule is more than happy to have modernized their workflows in the cloud, there are still some clients—and major clients at that—who require that contractors save final projects on tape for long-term storage. It no longer made sense to have staff trained on how to use the LTO system, so when a customer asked for a tape copy, they reached out to Backblaze for advice.

They needed a turnkey solution that they didn’t have to manage, and they definitely didn’t want to have to resort to reinvesting and managing tape hardware. Backblaze partner, TapeArk, fit the bill. TapeArk typically helps clients get data off of tape and into the cloud, but in this case they reversed the process. Molecule sent them a secure token to the exact piece of data they needed. TapeArk managed the download, put it on tape, and shipped it to the client.

If Molecule needs to send tape copies to clients in the future, they have an easy, hands-off solution and they don’t have to maintain an LTO system for infrequent use. Ben was grateful for the partnership and easy solution.

Cloud Workflows Free Up a Month of Time

Now that the staff no longer has to manage an LTO tape system, the team has recouped at least 30 payroll days a year that can be dedicated to supporting artists. Ben noted that with the workflows in the cloud, the nature of the IT workload has changed, and the team definitely appreciates having that time back to respond to changing demands.

Ready to move your VFX workflows to the cloud? Start testing today with 10GB of data storage free from Backblaze B2.

The post How to Run VFX Workflows in the Cloud appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Post Syndicated from Pat Patterson original https://www.backblaze.com/blog/building-a-multiregion-origin-store-with-backblaze-b2-fastly-computeedge/

Backblaze B2 Cloud Storage customers have long leveraged our partner Fastly’s Deliver@Edge CDN as an essential component of a modern, scalable web architecture. Complementing Deliver@Edge, Compute@Edge is a serverless computing environment built on the same caching platform to provide a general-purpose compute layer between the cloud and end users. Today, we’re excited to celebrate Fastly’s announcement of its Compute@Edge partner ecosystem.

Serverless computing is quickly gaining popularity among developers for its simplicity, agility, and functionality. In the serverless model, cloud providers allocate resources to applications on demand, managing the compute infrastructure on behalf of their customers. The term, “serverless,” is a little misleading: The servers are actually still there, but customers don’t have to get involved in their provisioning, configuration, maintenance, or scaling.

Fastly’s Compute@Edge represents the next generation of serverless computing—purpose-built for better performance, reduced latency, and enhanced visibility and security. Using Fastly’s tools, a developer can create an edge application, test it locally, then with one command, deploy it to the Compute@Edge platform. When a request for that application reaches any of Fastly’s global network of edge servers, the application is launched and running in microseconds and can instantly scale to tens of thousands of requests per second.

It’s difficult to overstate the power and flexibility this puts in your hands as a developer—your application can be running on every edge server, with access to every attribute of its incoming requests, assembling responses in any way you choose. For an idea of the possibilities, check out the Compute@Edge demos, in particular, the implementation of the video game classic, “Doom.”

We don’t have space in a single blog post to explore an edge application of that magnitude, but read on for a simple example of how you can combine Fastly’s Compute@Edge with Backblaze B2 to improve your website’s user experience, directing requests to the optimal origin store end point based on the user’s location.

The Case for a Multiregion Origin Store

Although the CDN caches resources to improve performance, if a requested resource is not present in the edge server cache, it must be fetched from the origin store. When the edge server is close to the origin store, the increase in latency is minimal. If, on the other hand, the edge server is on a different continent from the origin store, it can take significantly longer to retrieve uncached content. In most cases, this additional delay is hardly noticeable, but for websites with many resources that are frequently updated, it can add up to a sluggish experience for users. A solution is for the origin store to maintain multiple copies of a website’s content, each at an end point in a different region. This approach can dramatically reduce the penalty for cache misses, improving the user experience.

There is a problem here, though: How do we ensure that a given CDN edge server directs requests to the “best” end point? The answer: build an application that uses the edge server’s location to select the end point. I’ll explain how I did just that, creating a Fastly Compute@Edge application to proxy requests to Backblaze B2 buckets.

Creating an Application on Fastly Compute@Edge

The Fastly Compute@Edge developer documentation did a great job of walking me through creating a Compute@Edge application. As part of the process, I had to choose a starter kit—a simple working application targeting a specific use case. The Static Content starter kit was the ideal basis for my application—it demonstrates many useful techniques, such as generating an AWS V4 Signature and manipulating the request’s Host HTTP header to match the origin store.

The core of the application is just a few lines written in the Rust programming language:

#[fastly::main]
 
fn main(mut req: Request) -> Result<Response, Error> {
// 1. Where is the application running?
let pop = get_pop(&req);

// 9. Return the response to the client
return Ok(beresp);
}

In step one, the get_pop function returns the three-letter abbreviation for the edge server, or point of presence (POP). For the purposes of testing, you can specify a POP as a query parameter in your HTTP request. For example, https://three.interesting.words.edgecompute.app/image.png?pop=AMS will simulate the application running on the Amsterdam POP. Next, in step two, the application looks up the POP in a mapping of POPs to Backblaze B2 end points. There are about a hundred Fastly POPs spread around the world; I simply took the list generated by running the Fastly command-line tool with the POPs argument, and assigned POPs to Backblaze B2 end points based on their location:

• POPs in North America, South America, and Asia/Pacific map to the U.S. end point.
• POPs in Europe and Africa map to the EU end point.

I won’t step through the rest of the logic in detail here—the comments in the code sample above cover the basics; feel free to examine the code in detail on GitHub if you’d like a closer look.

Serve Your Own Data From Multiple Backblaze B2 Regions

As you can see in the screenshot above, Fastly has implemented a Deploy to Fastly button. You can use this to create your own copy of the Backblaze B2 Compute@Edge demo application in just a couple of minutes. You’ll need to gather a few prerequisites before you start:

• You must create Backblaze B2 accounts in both the U.S. and EU regions. If you have an existing account and you’re not sure which region it’s in, just take a look at the end point for one of your buckets. For example, this bucket is in the U.S. West region:

  

  To create your second account, go to the Sign Up page, and click the Region drop-down on the right under the big, red Sign Up button:

  

  Pick the region in which you don’t already have an account, and enter an email and password. Remember, your new account comes with 10GB of storage, free of charge, so there’s no need to enter your credit card details.

  Note: You’ll need to use a different email address from your existing account. If you don’t have a second email address, you can use the plus trick (officially known as sub-addressing) and reuse an existing email address. For example, if you used [email protected] for your existing B2 Cloud Storage account in the U.S. region, you can use [email protected] for your new EU account. Mail will be routed to the same inbox, and Backblaze B2 will be satisfied that it’s a different email address. This technique isn’t limited to Gmail, by the way, it works with many email providers.

• Create a private bucket in each account, and use your tool of choice to copy the same data into each of them. Make a note of the end point for each bucket.
• Create an application key with read access to each bucket.
• Sign up for a free Fastly account if you don’t already have one. Right now, this includes free credits for Compute@Edge.
• Sign up for a free GitHub account.
• Go to the Backblaze B2/Fastly Compute@Edge Demo GitHub repository, click the Deploy to Fastly button, and follow the prompts. The repository will be forked to your GitHub account and then deployed to Fastly.
• Important: There is one post-deploy step you must complete before your application will work! In your new GitHub repository, navigate to src/config.rs and hit the pencil icon near the top right to edit the file. Change the origin configuration in lines 18-31 to match your buckets and their end points. Alternatively, you can, of course, clone the repository to your local machine, edit it there, and push the changes back to GitHub.

Once you have your accounts and buckets created, it takes just a few minutes to deploy the application. Watch me walk through the process:

What Can You Do With Fastly’s Compute@Edge and Backblaze B2?

My simple demo application only scratches the surfaces of Compute@Edge. How could you combine Fastly’s edge computing platform with Backblaze B2 to create a new capability for your website? Check out Fastly’s collection of over 100 Compute@Edge code samples for inspiration. If you come up with something neat and share it on GitHub, let me know in the comments and I’ll round up a bundle of Backblaze-branded goodies, just for you!

The post Building a Multiregion Origin Store With Backblaze B2 + Fastly Compute@Edge appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.