All posts by Skip Levens

Digital Nomad: Amphibious Filmmaker Chris Aguilar

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/digital-nomad-amphibious-filmmaker-chris-aguilar/

Map showing Digital Nomad Chris Aguilar at Work in the Ka'iwi Channel

The Digital Nomads Series

In this blog series, we explore how you can master the nomadic life—whether for a long weekend, an extended working vacation, or maybe even the rest of your career. We profile professionals we’ve met who are stretching the boundaries of what (and where) an office can be, and glean lessons along the way to help you to follow in their footsteps. In our first post in the series, we provided practical tips for working on the road. In this edition, we profile Chris Aguilar, Amphibious Filmmaker.


There are people who do remote filming assignments, and then there’s Chris, the Producer/Director of Fin Films. For him, a normal day might begin with gathering all the equipment he’ll need—camera, lenses, gear, cases, batteries, digital storage—and securing it in a waterproof Pelican case which he’ll then strap to a paddleboard for a long swim to a race boat far out on the open ocean.

This is because Chris, a one-man team, is the preeminent cinematographer of professional paddleboard racing. When your work day involves operating from a beachside hotel, and being on location means bouncing up and down in a dinghy some 16 miles from shore, how do you succeed? We interviewed Chris to figure out.

Chris filming on the water during Molokai 2 Oahu Paddleboard World Championships
Chris filming on the water during Molokai 2 Oahu Paddleboard World Championships — Photo by Troy Nebecker

Getting Ready for a Long Shoot

To save time in the field, Chris does as much prep work as he can. Knowing that he needs to be completely self-sufficient all day—he can’t connect to power or get additional equipment—he gathers and tests all of the cameras he’ll need for all the possible shots that might come up, packs enough SD camera cards, and grabs an SSD external drive large enough to store an entire day’s footage.

Chris edits in Adobe Premiere, so he preloads a template on his MacBook Pro to hold the day’s shots and orders everything by event so that he can drop his content in and start editing it down as quickly as possible. Typically, he chooses a compatible format that can hold all of the different content he’ll shoot. He builds a 4K timeline at 60 frames per second that can take clips from multiple cameras yet can export to other sizes and speeds as needed for delivery.

Map of the Molokai 2 Oahu Paddleboard Route
The Molokai 2 Oahu Paddleboard Race Route

Days in the Life

Despite being in one of the most exotic and glamorous locations in the world (Hawaii), covering a 32-mile open-ocean race is grueling. Chris’s days start as early as 5AM with him grabbing shots as contestants gather, then filming as many as 35 interviews on race-day eve. He does quick edits of these to push content out as quickly as possible for avid fans all over the world.

The next morning, before race time, he double-checks all the equipment in his Pelican case, and, when there’s no dock, he swims out to the race- or camera boat. After that, Chris shoots as the race unfolds, constantly swapping out SD cards. When he’s back on dry land his first order of business is copying over all of the content to his external SSD drive.

Even after filming the race’s finish, awards ceremonies, and wrap-up interviews, he’s still not done: By 10PM he’s back at the hotel to cut a highlight reel of the day’s events and put together packages that sports press can use, including the Australian press that needs content for their morning sports shows.

For streaming content in the field, Chris relies on Google Fi through his phone because it can piggyback off of a diverse range of carriers. His backup network solution is a Verizon hotspot that usually covers him where Google Fi cannot. For editing and uploading, he’s found that he can usually rely on his hotel’s network. When that doesn’t work, he defaults to his hotspot, or a coffee shop. (His pro tip is that, for whatever reason, the Starbucks in Hawaii typically have great internet.)

A paddleboarder

Building a Case

After years of shooting open-ocean events, Chris has settled on a tried and true combination of gear—and it all fits in a single, waterproof Pelican 1510 case. His kit has evolved to be as simple and flexible as possible, allowing him to cover multiple shooting roles in a hostile environment including sand, extreme sun-glare on the water, haze, fog, and of course, the ever-present ocean water.

At the same time, his gear needs to accommodate widely varied shooting styles: Chris needs to be ready to capture up close and personal interviews; wide, dramatic shots of the pre-race ceremonies; as well as a combination of medium shots of several racers on the ocean and long, telephoto shots of individuals—all from a moving boat bobbing on the ocean. Here’s his “Waterproof Kit List”:

The Case
Pelican 1510

Pelican 1510 Waterproof Case

The Cameras

Chris likes compact, rugged camcorders from Panasonic. They have extremely long battery life, and the latest generation have large sensor sizes, wide dynamic range and even built-in ND filter wheels to compensate for the glare on the water. He’ll also bring other cameras for special shots, like an 8mm film camera for artistic shots, or a GoPro for the classic ‘from under the sea to the waterline’ shots.

Primary Interview Camera

  • Panasonic EVA1 5.7K Compact Cinema Camcorder 4K 10b 4:2:2 with EF lens-mount (with rotating lens kit depending on the event)

Panasonic EVA1 5.7K Compact Cinema Camcorder 4K 10b 4:2:2 with EF lens-mount

Action Camera and B-Roll

  • Panasonic AG-CX350 (or EVA1 kitted out similarly if the CX350 isn’t available)

Panasonic AG-CX350 Camera

Stills and Video

 

  • Panasonic GH5 20.3MP and 4K 60fps 4:2:2 10-b Mirrorless ILC camera

    Panasonic GH5 20.3MP and 4K 60fps 4:2:2 10-b Mirrorless ILC camera

 

Special Purpose and B-Roll Shots

  • Eumig Nautica Super 8 film self-sealed waterproof camera

    Eumig Nautica Super 8 film self-sealed waterproof camera

  • 4K GoPro in a waterproof dome housing

    4K GoPro in a waterproof dome housing

    Storage

    As a one-person show, Chris invests in enough SD cards for his cameras that can cover the entire day’s shooting without having to reuse cards. Chris will then copy all of those card’s content to a bus-powered SSD drive.

  • 8-12 64GB or 128GB SD cards

    Panasonic SD Card

  • 1 TB SSD Glyph or G-Tech SSD drive

    1 TB SSD Glyph drive

    Other Equipment

    • Multiple Neutral Density filters. These filters reduce the intensity of all wavelengths without affecting color. With ND filters the operator can dial in combinations of aperture, exposure time and sensor sensitivity without being overexposed, and delivers more ‘filmic’ looks, setting the aperture to a low value for sharper images, or wide open for a shallow depth-of-field
    • Extra batteries. Needless to say having extra batteries for his cameras and his phone is critical when he may not be able to recharge for 12 hours or more.

    Now, The Real Work Begins

    When wrapping up an event’s coverage, all of the content captured needs to be stored and managed. Chris’s previous workflow required transferring the raw and finished files to external drives for storage. That added up to a lot of drives. Chris estimates that over the years he had stored about 20 terabytes of footage on paddleboarding alone.

    Managing all those drives proved to be too big of a task for someone who is rarely in his production office. Chris needed access to his files from wherever he was, and a way to view, catalog, and share the content with collaborators.

    As he got his approach dialed to accommodate remote broadband speed, storage drive wrangling, inexpensive cloud storage, and cloud-based digital asset management systems, putting all his content into the cloud became an option for Chris. Using Backblaze’s B2 Cloud Storage along with iconik content management software, what used to take several days in the office searching through hard drives for specific footage to edit or share with a collaborator now involves just a few keyword searches and a matter of minutes to share via iconik.

    For a digital media nomad like Chris, digitally native solutions based in the cloud make a lot of sense. Plus, Chris knows that the content is safely and securely stored, and not exposed to transport challenges, accidents (including those involving water), and other difficulties that could spoil both his day and that of his clients.

    Learn More About How Chris Works Remotely

    You can learn more about Chris, Fin Film Company, and how he works from the road in our case study on Fin Films. We’ve also linked to Chris’s Kit Page for those of you who just can’t get enough of this gear…


    We’d Love to Hear Your Digital Nomad Stories

    If you consider yourself a digital nomad and have an interesting story about using Backblaze Cloud Backup or B2 Cloud Storage from the road (or wherever), we’d love to hear about it, and perhaps feature your story on the blog. Tell us what you’ve been doing on the road at mailbag@backblaze.com.

    You can view all the posts in this series on the Digital Nomads page in our Blog Archives.

The post Digital Nomad: Amphibious Filmmaker Chris Aguilar appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The Shocking Truth — Managing for Hard Drive Failure and Data Corruption

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/managing-for-hard-drive-failures-data-corruption/

hard disk drive covered in 0s, 1s, ?s

Ah, the iconic 3.5″ hard drive, now approaching a massive 16TB of storage capacity. Backblaze storage pods fit 60 of these drives in a single pod, and with well over 750 petabytes of customer data under management in our data centers, we have a lot of hard drives under management.

Yet most of us have just one, or only a few of these massive drives at a time storing our most valuable data. Just how safe are those hard drives in your office or studio? Have you ever thought about all the awful, terrible things that can happen to a hard drive? And what are they, exactly?

It turns out there are a host of obvious physical dangers, but also other, less obvious, errors that can affect the data stored on your hard drives, as well.

Dividing by One

It’s tempting to store all of your content on a single hard drive. After all, the capacity of these drives gets larger and larger, and they offer great performance of up to 150 MB/s. It’s true that flash-based hard drives are far faster, but the dollars per gigabyte price is also higher, so for now, the traditional 3.5″ hard drive holds most data today.

However, having all of your precious content on a single, spinning hard drive is a true tightrope without a net experience. Here’s why.

Drivesaver Failure Analysis by the Numbers

Drive failures by possible external force

I asked our friends at Drivesavers, specialists in recovering data from drives and other storage devices, for some analysis of the hard drives brought into their labs for recovery. What were the primary causes of failure?

Reason One: Media Damage

The number one reason, accounting for 70 percent of failures, is media damage, including full head crashes.

Modern hard drives stuff multiple, ultra thin platters inside that 3.5 inch metal package. These platters spin furiously at 5400 or 7200 revolutions per minute — that’s 90 or 120 revolutions per second! The heads that read and write magnetic data on them sweep back and forth only 6.3 micrometers above the surface of those platters. That gap is about 1/12th the width of a human hair and a miracle of modern technology to be sure. As you can imagine, a system with such close tolerances is vulnerable to sudden shock, as evidenced by Drivesavers’ results.

This damage occurs when the platters receive shock, i.e. physical damage from impact to the drive itself. Platters have been known to shatter, or have damage to their surfaces, including a phenomenon called head crash, where the flying heads slam into the surface of the platters. Whatever the cause, the thin platters holding 1s and 0s can’t be read.

It takes a surprisingly small amount of force to generate a lot of shock energy to a hard drive. I’ve seen drives fail after simply tipping over when stood on end. More typically, drives are accidentally pushed off of a desktop, or dropped while being carried around.

A drive might look fine after a drop, but the damage may have been done. Due to their rigid construction, heavy weight, and how often they’re dropped on hard, unforgiving surfaces, these drops can easily generate the equivalent of hundreds of g-forces to the delicate internals of a hard drive.

To paraphrase an old (and morbid) parachutist joke, it’s not the fall that gets you, it’s the sudden stop!

Reason Two: PCB Failure

The next largest cause is circuit board failure, accounting for 18 percent of failed drives. Printed circuit boards (PCBs), those tiny green boards seen on the underside of hard drives, can fail in the presence of moisture or static electric discharge like any other circuit board.

Reason Three: Stiction

Next up is stiction (a portmanteau of friction and sticking), which occurs when the armatures that drive those flying heads actually get stuck in place and refuse to operate, usually after a long period of disuse. Drivesavers found that stuck armatures accounted for 11 percent of hard drive failures.

It seems counterintuitive that hard drives sitting quietly in a dark drawer might actually contribute to its failure, but I’ve seen many older hard drives pulled from a drawer and popped into a drive carrier or connected to power just go thunk. It does appear that hard drives like to be connected to power and constantly spinning and the numbers seem to bear this out.

Reason Four: Motor Failure

The last, and least common cause of hard drive failure, is hard drive motor failure, accounting for only 1 percent of failures, testament again to modern manufacturing precision and reliability.

Mitigating Hard Drive Failure Risk

So now that you’ve seen the gory numbers, here are a few recommendations to guard against the physical causes of hard drive failure.

1. Have a physical drive handling plan and follow it rigorously

If you must keep content on single hard drives in your location, make sure your team follows a few guidelines to protect against moisture, static electricity, and drops during drive handling. Keeping the drives in a dry location, storing the drives in static bags, using static discharge mats and wristbands, and putting rubber mats under areas where you’re likely to accidentally drop drives can all help.

It’s worth reviewing how you physically store drives, as well. Drivesavers tells us that the sudden impact of a heavy drawer of hard drives slamming home or yanked open quickly might possibly damage hard drives!

2. Spread failure risk across more drives and systems

Improving physical hard drive handling procedures is only a small part of a good risk-reducing strategy. You can immediately reduce the exposure of a single hard drive failure by simply keeping a copy of that valuable content on another drive.This is a common approach for videographers moving content from cameras shooting in the field back to their editing environment. By simply copying content over from one fast drive to another, the odds of both drives failing at once are less likely. This is certainly better than keeping content on only a single drive, but definitely not a great long-term solution.

Multiple drive NAS and RAID systems reduce the impact of failing drives even further. A RAID 6 system composed of eight drives not only has much faster read and write performance than a single drive, but two of its drives can fail and still serve your files, giving you time to replace those failed drives.

Mitigating Data Corruption Risk

The Risk of Bit Flips

Beyond physical damage, there’s another threat to the files stored on hard disks: small, silent bit flip errors often called data corruption or bit rot.

Bit rot errors occur when individual bits in a stream of data in files change from one state to another (positive or negative, 0 to 1, and vice versa). These errors can happen to hard drive and flash storage systems at rest, or be introduced as a file is copied from one hard drive to another.

While hard drives automatically correct single-bit flips on the fly, larger bit flips can introduce a number of errors. This can either cause the program accessing them to halt or throw an error, or perhaps worse, lead you to think that the file with the errors is fine!

Bit Flip Errors by the Book

In a landmark study of data failures in large systems, Disk failures in the real world:
What does an MTTF of 1,000,000 hours mean to you?
, Bianca Schroeder and Garth A. Gibson reported that “a large number of the problems attributed to CPU and memory failures were triggered by parity errors, i.e. the number of errors is too large for the embedded error correcting code to correct them.”

Flash drives are not immune either. Bianca Shroeder recently published a similar study of flash drives, Flash Reliability in Production: The Expected and the Unexpected, and found that “…between 20-63% of drives experienced at least one of the (unrecoverable read errors) during the time it was in production. In addition, between 2-6 out of 1,000 drive days were affected.”

“These UREs are almost exclusively due to bit corruptions that ECC cannot correct. If a drive encounters a URE, the stored data cannot be read. This either results in a failed read in the user’s code, or if the drives are in a RAID group that has replication, then the data is read from a different drive.”

Exactly how prevalent bit flips are is a controversial subject, but if you’ve ever retrieved a file from an old hard drive or RAID system and see sparkles in video, corrupt document files, or lines or distortions in pictures, you’ve seen the results of these errors.

Protecting Against Bit Flip Errors

There are many approaches to catching and correcting bit flip errors. From a system designer standpoint they usually involve some combination of multiple disk storage systems, multiple copies of content, data integrity checks and corrections, including error-correcting code memory, physical component redundancy, and a file system that can tie it all together.

Backblaze has built such a system, and uses a number of techniques to detect and correct file degradation due to bit flips and deliver extremely high data durability and integrity, often in conjunction with Reed-Solomon erasure codes.

Thanks to the way object storage and Backblaze B2 works, files written to B2 are always retrieved exactly as you originally wrote them. If a file ever changes from the time you’ve written it, say, due to bit flip errors, it will either be reproduced from a redundant copy of your file, or even mathematically reconstructed with erasure codes.

So the simplest, and certainly least expensive way to get bit flip protection for the content sitting on your hard drives is to simply have another copy on cloud storage.

Resources:

The Ideal Solution — Performance and Protection

With some thought, you can apply these protection steps to your environment and get the best of both worlds: the performance of your content on fast, local hard drives, and the protection of having a copy on object storage offsite with the ultimate data integrity.

The post The Shocking Truth — Managing for Hard Drive Failure and Data Corruption appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The Profound Benefits of Cloud Collaboration for Business Users

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/cloud-collaboration-for-business-users/

The Profound Benefits of Cloud Collaboration for Business Users

Apple’s annual WWDC is highlighting high-end desktop computing, but it’s laptop computers and the cloud that are driving a new wave of business and creative collaboration

WWDC, Apple’s annual megaconference for developers kicks off this week, and Backblaze has team members on the ground to bring home insights and developments. Yet while everyone is drooling over the powerful new Mac Pro, we know that the majority of business users use a portable computer as their primary system for business and creative use.

The Rise of the Mobile, Always On, Portable Workstation

Analysts confirm this trend towards the use of portable computers and the cloud. IDC’s 2019 Worldwide Quarterly Personal Computing Device Tracker report shows that desktop form-factor systems comprise only 22.6% of new systems and laptops and portables are chosen almost twice as much at 42.4%.

After all, these systems are extremely popular with users and the DevOps and IT teams that support them. Small and self-contained, with massive compute power, modern laptops have fast SSD drives and always-connected Wi-Fi, helping users be productive anywhere: in the field, on business trips, and at home. Surprisingly, companies today can deploy massive fleets of these notebooks with extremely lean staff. At the inaugural MacDevOps conference a few years ago Google’s team shared that they managed 65,000 Macs with a team of seven admins!

Laptop Backup is More Important Than Ever

With the trend towards leaner IT staffs, and the dangers of computers in the field being lost, dropped or damaged, having a reliable backup system that just works is critical. Despite the proliferation of teams using shared cloud documents and email, all of the other files on your laptop you’re working on — the massive presentation due next week or the project that’s not quite ready to share on Google Drive — all have no protection without backup, which is of course why Backblaze exists!

Cloud as a Shared Business Content Hub is Changing Everything

When a company is backing up users’ files comfortably to the cloud, the next natural step is to adopt cloud-based storage like Backblaze B2 for your teams. With over 750 petabytes of customer data under management, Backblaze has worked with businesses of every size as they adopt cloud storage. Each customer and business does so for different reasons.

In the past, a business department typically would get a share of a company’s NAS server and was asked to keep all of the department’s shared documents there. But outside the corporate firewall, it turns out these systems are hard to access remotely from the road. They require VPNs and a constant network connection to mount a corporate shared drive via SMB or NFS. And, of course, running out of space and storing large files was an ever present problem.

Sharing Business Content in the Cloud Can be Transformational for Businesses

When considering a move to cloud-based storage for your team, some benefits seem obvious, but others are more profound and show that cloud storage is emerging as a powerful, organizing platform for team collaboration.

Shifting to cloud storage delivers these well-known benefits:

  • Pay only for storage you actually need
  • Grow as large and as quickly as you might need
  • Service, management, and upgrades are built in to the service
  • Pay for service as you use it out of operating expenses vs. onerous capital expenses

But shifting to shared, cloud storage yields even more profound benefits:

Your Business Content is Easier to Organize and Manage: When your team’s content is in one place, it’s easier to organize and manage, and users can finally let go of stashing content all over your organization or leaving it on their laptops. All of your tools to mine and uncover your business’s content work more efficiently, and your users do as well.

You Get Simple Workflow Management Tools for Free: Storage can fit your business processes much easier with cloud storage and do it on the fly. If you ever need to set up separate storage for teams of users, or define read/write rules for specific buckets of content, it’s easy to configure with cloud storage.

You Can Replace External File-Sharing Tools: Since most email services balk at sending large files, it’s common to use a file sharing service to share big files with other users on your team or outside your organization. Typically this means having to download a massive file, re-upload it to a file-sharing service, and publish that file-sharing link. When your files are already in cloud, sharing it is as simple as retrieving a URL location.

In fact, this is exactly how Backblaze organizes and serves PDF content on our website like customer case studies. When you click on a PDF link on the Backblaze website, it’s served directly from one of these links from a B2 bucket!

You Get Instant, Simple Policy Control over Your Business or Shared Content: B2 offers simple-to-use tools to keep every version of a file as it’s created, keep just the most recent version, or choose how many versions you require. Want to have your shared content links time-out after a day or so? This and more is all easily done from your B2 account page:

B2 Lifecycle Settings
An example of setting up shared link rules for a time-sensitive download: The file is available for 3 days, then deleted after 10 days

You’re One Step Away from Sharing That Content Globally: As you can see, beyond individual file-sharing, cloud storage like Backblaze B2 can serve as your origin store for your entire website. With the emergence of content delivery networks (CDN), you’re now only a step away from sharing and serving your content globally.

To make this easier, Backblaze joined the Bandwidth Alliance, and offers no-cost egress from your content in Backblaze B2 to Cloudflare’s global content delivery network.

Customers that adopt this strategy can dramatically slash the cost of serving content to their users.

"The combination of Cloudflare and Backblaze B2 Cloud Storage saves Nodecraft almost 85% each month on the data storage and egress costs versus Amazon S3." - James Ross, Nodecraft Co-founder/CTO

Read the Nodecraft/Backblaze case study.

Get Sophisticated Content Discovery and Compliance Tools for Your Business Content: With more and more business content in cloud storage, finding the content you need quickly across millions of files, or surfacing content that needs special storage consideration (for GDPR or HIPAA compliance, for example) is critical.

Ideally, you could have your own private, customized search engine across all of your cloud content, and that’s exactly what a new class of solutions provide.

With Acembly or Aparavi on Backblaze, you can build content indexes and offer deep search across all of your content, and automatically apply policy rules for management and retention.

Where Are You in the Cloud Collaboration Trend?

The trend to mobile, always-on workers building and sharing ever more sophisticated content around cloud storage as a shared hub is only accelerating. Users love the freedom to create, collaborate and share content anywhere. Businesses love the benefits of having all of that content in an easily managed repository that makes their entire business more flexible and less expensive to operate.

So, while device manufacturers like Apple may announce exciting Pro level workstations, the need for companies and teams to collaborate and be effective on the move is an even more important and compelling issue than ever before. The cloud is an essential element of that trend that can’t be underestimated.

•  •  •

Upcoming Free Webinars

Wednesday, June 5, 10am PT
Learn how Nodecraft saved 85% on their cloud storage bill with Backblaze B2 and Cloudflare.
Join the Backblaze/Nodecraft webinar.

Thursday, June 13, 10am PT
Want to learn more about turning content in Backblaze B2 into searchable content with powerful policy rules?
Join the Backblaze/Aparavi webinar
.

The post The Profound Benefits of Cloud Collaboration for Business Users appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Migrating 23TB from Amazon S3 to Backblaze B2 in Just Seven Hours

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/migrating-23tb-from-amazon-s3-to-backblaze-b2-in-just-seven-hours/

flowchart of data transfer - Cloudflare - Bandwidth Alliance FTW! - Backblaze B2 Cloud Storage - Free Bandwidth - Nodecraft

Like many Backblaze customers, Nodecraft realized they could save a fortune by shifting their cloud storage to Backblaze and invest it elsewhere in growing their business. In this post that originally appeared on Nodecraft’s blog, Gregory R. Sudderth, Nodecraft’s Senior DevOps Engineer, shares the steps they took to first analyze, test, and then move that storage.
— Skip Levens

Overview

TL;DR: Nodecraft moved 23TB of customer backup files from AWS S3 to Backblaze B2 in just 7 hours.

Nodecraft.com is a multiplayer cloud platform, where gamers can rent and use our servers to build and share unique online multiplayer servers with their friends and/or the public. In the course of server owners running their game servers, there are backups generated including the servers’ files, game backups and other files. It goes without saying that backup reliability is important for server owners.

In November 2018, it became clear to us at Nodecraft that we could improve our costs if we re-examine our cloud backup strategy. After looking at the current offerings, we decided we were moving our backups from Amazon’s S3 to Backblaze’s B2 service. This article describes how our team approached it, why, and what happened, specifically so we could share our experiences.

Benefits

Due to S3 and B2 being at least nearly equally* accessible, reliable, available, as well as many other providers, our primary reason for moving our backups now became pricing. As we started into the effort, other factors such as variety of API, quality of API, real-life workability, and customer service started to surface.

After looking at a wide variety of considerations, we decided on Backblaze’s B2 service. A big part of the costs of this operation is their bandwidth, which is amazingly affordable.

The price gap between the two object storage systems come from the Bandwidth Alliance between Backblaze and Cloudflare, a group of providers that have agreed to not charge (or heavily discount) for data leaving inside the alliance of networks (“egress” charges). We at Nodecraft use Cloudflare extensively and so this left only the egress charges from Amazon to Cloudflare to worry about.

In normal operations, our customers both constantly make backups as well as access them for various purposes and there has been no change to their abilities to perform these operations compared to the previous provider.

Considerations

As with any change in providers, the change-over must be thought out with great attention to detail. When there were no quality issues previously and circumstances are such that a wide field of new providers can be considered, the final selection must be carefully evaluated. Our list of concerns included these:

  • Safety: we needed to move our files and ensure they remain intact, in a redundant way
  • Availability: the service must both be reliable but also widely available ** (which means we needed to “point” at the right file after its move, during the entire process of moving all the files: different companies have different strategies, one bucket, many buckets, regions, zones, etc)
  • API: we are experienced, so we are not crazy about proprietary file transfer tools
  • Speed: we needed to move the files in bulk and not brake on rate limitations, and…

…improper tuning could turn the operation into our own DDoS.

All these factors individually are good and important, but when crafted together, can be a significant service disruption. If things can move easily, quickly, and, reliably, improper tuning could turn the operation into our own DDoS. We took thorough steps to make sure this wouldn’t happen, so an additional requirement was added:

Tuning: Don’t down your own services, or harm your neighbors

What this means to the lay person is “We have a lot of devices in our network, we can do this in parallel. If we do it at full-speed, we can make our multiple service providers not like us too much… maybe we should make this go at less than full speed.”

Important Parts

To embrace our own cloud processing capabilities, we knew we would have to take a two tier approach in both the Tactical (move a file) and Strategic (tell many nodes to move all the files) levels.

Strategic

Our goals here are simple: we want to move all the files, move them correctly, and only once, but also make sure operations can continue while the move happens. This is key because if we had used one computer to move the files, it would take months.

The first step to making this work in parallel was to build a small web service to allow us to queue a single target file to be moved at a time to each worker node. This service provided a locking mechanism so that the same file wouldn’t be moved twice, both concurrently or eventually. The timer for the lock to expire (with error message) was set to a couple hours. This service was intended to be accessed via simple tools such as curl.

We deployed each worker node as a Docker container, spread across our Docker Swarm. Using the parameters in a docker stack file, we were able to define how many workers per node joined the task. This also ensured more expensive bandwidth regions like Asia Pacific didn’t join the worker pool.

Tactical

Nodecraft has multiple fleets of servers spanning multiple datacenters, and our plan was to use spare capacity on most of them to move the backup files. We have experienced a consistent pattern of access of our servers by our users in the various data centers across the world, and we knew there would be availability for our file moving purposes.

Our goals in this part of the operation are also simple, but have more steps:

  • Get the name/ID/URL of a file to move which…
    • locks the file, and…
    • starts the fail timer
  • Get the file info, including size
  • DOWNLOAD: Copy the file to the local node (without limiting the node’s network availability)
  • Verify the file (size, ZIP integrity, hash)
  • UPLOAD: Copy the file to the new service (again without impacting the node)
  • Report “done” with new ID/URL location information to the Strategic level, which…
    • …releases the lock in the web service, cancels the timer, and marks the file DONE
diagram of Nodecraft data migration from AWS S3 to Backblaze B2 Cloud Storage
Diagram illustrating how the S3 to B2 move was coordinated

The Kill Switch

In the case of a potential run-away, where even the in-band Docker Swarm commands themselves, we decided to make sure we had a kill switch handy. In our case, it was our intrepid little web service–we made sure we could pause it. Looking back, it would be better if it used a consumable resource, such as a counter, or a value in a database cell. If we didn’t refresh the counter, then it would stop all its own. More on “runaways” later.

Real Life Tuning

Our business has daily, weekly, and other cycles of activity that are predictable. Most important is our daily cycle, that trails after the Sun. We decided to use our nodes that were in low-activity areas to carry the work, and after testing, we found that if we tune correctly this doesn’t affect the relatively light loads of the servers in that low-activity region. This was backed up by verifying no change in customer service load using our metrics and those of our CRM tools. Back to tuning.

Initially we tuned the DOWN file transfer speed equivalent to 3/4ths of what wget(1) could do. We thought “oh, the network traffic to the node will fit in-between this so it’s ok”. This is mostly true, but only mostly. This is a problem in two ways. The cause of the problems is that isolated node tests are just that—isolated. When a large number of nodes in a datacenter are doing the actual production file transfers, there is a proportional impact that builds as the traffic is concentrated towards the egress point(s).

Problem 1: you are being a bad neighbor on the way to the egress points. Ok, you say “well we pay for network access, let’s use it” but of course there’s only so much to go around, but also obviously “all the ports of the switch have more bandwidth than the uplink ports” so of course there will be limits to be hit.

Problem 2: you are being your own bad neighbor to yourself. Again, if you end-up with your machines being network-near to each other in a network-coordinates kind of way, your attempts to “use all that bandwidth we paid for” will be throttled by the closest choke point, impacting only or nearly only yourself. If you’re going to use most of the bandwidth you CAN use, you might as well be mindful of it and choose where you will put the chokepoint, that the entire operation will create. If one is not cognizant of this concern, one can take down entire racks of your own equipment by choking the top-of-rack switch, or, other networking.

By reducing our 3/4ths-of-wget(1) tuning to 50% of what wget could do for a single file transfer, we saw our nodes still functioning properly. Your mileage will absolutely vary, and there’s hidden concerns in the details of how your nodes might or might not be near each other, and their impact on hardware in between them and the Internet.

Old Habits

Perhaps this is an annoying detail: Based on previous experience in life, I put in some delays. We scripted these tools up in Python, with a Bourne shell wrapper to detect fails (there were) and also because for our upload step, we ended up going against our DNA and used the Backblaze upload utility. By the way, it is multi-threaded and really fast. But in the wrapping shell script, as a matter of course, in the main loop, that was first talking to our API, I put in a sleep 2 statement. This creates a small pause “at the top” between files.

This ended up being key, as we’ll see in a moment.

How It (The Service, Almost) All Went Down

What’s past is sometimes not prologue. Independent testing in a single node, or even a few nodes, was not totally instructive to what really was going to happen as we throttled up the test. Now when I say “test” I really mean, “operation”.

Our initial testing was concluded “Tactically” as above, for which we used test files, and were very careful in the verification thereof. In general, we were sure that we could manage copying a file down (Python loop) and verifying (unzip -T) and operate the Backblaze b2 utility without getting into too much trouble…but it’s the Strategic level that taught us a few things.

Remembering to a foggy past where “6% collisions on a 10-BASE-T network and its game over”…yeah that 6%. We throttled up the number of replicas in the Docker Swarm, and didn’t have any problems. Good. “Alright.” Then we moved the throttle so to speak, to the last detent.

We had nearly achieved self-DDoS.

It wasn’t all that bad, but, we were suddenly very, very happy with our 50%-of-wget(1) tuning, and our 2 second delays between transfers, and most of all, our kill switch.

Analysis

TL;DR — Things went great.

There were a couple files that just didn’t want to transfer (weren’t really there on S3, hmm). There were some DDoS alarms that tripped momentarily. There was a LOT of traffic…and, then, the bandwidth bill.

Your mileage may vary, but there’s some things to think about with regards to your bandwidth bill. When I say “bill” it’s actually a few bills.

diagram of bandwidth data flow savings switching away from AWS S3 to Backblaze B2 cloud storage
Cloudflare Bandwidth Alliance cost savings

As per the diagram above, moving the file can trigger multiple bandwidth charges, especially as our customers began to download the files from B2 for instance deployment, etc. In our case, we now only had the S3 egress bill to worry about. Here’s why that works out:

  • We have group (node) discount bandwidth agreements with our providers
  • B2 is a member of the Bandwidth Alliance…
  • …and so is Cloudflare
  • We were accessing our S3 content through our (not free!) Cloudflare account public URLs, not by the (private) S3 URLs.

Without saying anything about our confidential arrangements with our service partners, the following are both generally true: you can talk to providers and sometimes work out reductions. Also, they especially like it when you call them (in advance) and discuss your plans to run their gear hard. For example, on another data move, one of the providers gave us a way to “mark” our traffic a certain way, and it would go through a quiet-but-not-often-traveled part of their network; win win!

Want More?

Thanks for your attention, and good luck with your own byte slinging.

by Gregory R. Sudderth
Nodecraft Senior DevOps Engineer

* Science is hard, blue keys on calculators are tricky, and we don’t have years to study things before doing them

Free Webinar
Nodecraft’s Data Migration From S3 to B2

Wednesday, June 5, 2019 at 10am PT
Cloud-Jitsu: Migrating 23TB from AWS S3 to Backblaze B2 in 7 hours

The post Migrating 23TB from Amazon S3 to Backblaze B2 in Just Seven Hours appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze’s Must See List for NAB 2019

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/what-not-to-miss-nab2019/

Collage of logos from Backblaze B2 cloud storage partners

With NAB 2019 only days away, the Backblaze team is excited to launch into the world’s largest event for creatives, and our biggest booth yet!

Must See — Backblaze Booth

This year we’ll be celebrating some of the phenomenal creative work by our customers, including American Public Television, Crisp Video, Falcons’ Digital Creative, WunderVu, and many more.

We’ll have workflow experts standing by to chat with you about your workflow frustrations, and how Backblaze B2 Cloud Storage can be the key to unlocking efficiency and solving storage challenges throughout your entire workflow: From Action! To Archive. With B2, you can focus on creating and managing content, not managing storage.

Create: Bring Your Story to Life

Stop by our booth and we can show you how you can protect your content from ingest through work-in-process by syncing seamlessly to the cloud. We can also detail how you can improve team collaboration and increase content reuse by organizing your content with one of our MAM integrations.

Distribute: Share Your Story With the World

Our experts can show you how B2 can help you scale your content library instantly and indefinitely, and avoid the hassle and expense of on-premises storage. We can demonstrate how everything in your content library can be served directly from your B2 account or through our content delivery partners like Cloudflare.

Preserve: Make Sure Your Story Lives Forever

Want to see the math behind the first cloud storage that’s more affordable than LTO? We can step through the numbers. We can also show you how B2 will keep your archived content accessible, anytime, and anywhere, through a web browser, API calls, or one of our integrated applications listed below.

Must See — Workflow Integrations You Can Count On

Our fantastic workflow partners are a critical part of your creative workflow backed by Backblaze — and there’s a lot of partner news to catch up on!

Drop by our booth to pick up a handy map to help you find Backblaze partners on the show floor including:

Backup and Archive Workflow Integrations

Archiware P5, booth SL15416
SyncBackPro, Wynn Salon — J

File Transfer Acceleration, Data Wrangling, Data Movement

FileCatalyst, booth SL12116
Hedge, booth SL14805

Asset and Collaboration Managers

axle ai, booth SL15116
Cantemo iconik, booth SL6021
Cantemo (Portal), booth SL6021
CatDV, booth SL5421
Cubix (Ortana Media Group), booth SL5922
eMAM, booth SL10224

Workflow Storage

Facilis, booth SL6321
GB Labs, booth SL5324
ProMAX, booth SL6313
Scale Logic, booth SL11109
Tiger Technology, booth SL8505
QNAP, booth SL15716
Seagate, booth SL8511
StorageDNA, booth SL11810

Must See — Backblaze Events during NAB

Monday morning we’re delivering a presentation in the Scale Logic Knowledge Zone, and Tuesday night of NAB we’re honored to help sponsor the all-new Faster Together event that replaces the long-standing Las Vegas Creative User Supermeet event.

We’ll be raffling off a Hover2 4K drone powered by AI to help you get that perfect drone shot for your next creative film! So after the NAB show wraps up on Tuesday, head over to the Rio main ballroom for a night of mingling with creatives and amazing talks by some of the top editors, colorists, and VFX artists in the industry.

ProVideoTech and Backblaze at Scale Logic Knowledge Zone
Monday April 8 at 11 AM
Scale Logic Knowledge Zone, NAB Booth SL111109
Monday of NAB, Backblaze and PVT will deliver a live presentation for NAB attendees on how to build hybrid-cloud workflows with Cantemo and Backblaze.
Scale Logic Media Management Knowledge Zone

Backblaze at The Faster Together Stage
Tuesday, April 9
Rio Las Vegas Hotel and Casino
Doors open at 4:30 PM, stage presentations begin at 7:00 PM
Reserve Tickets for the Faster Together event

If you haven’t yet, be sure to sign up and reserve your meeting time with the Backblaze team, and add us to your Map My Show NAB plan and we’ll see you there!

  NAB 2019 is just a few days away. NABShow logoSchedule a meeting with our cloud storage experts to learn how B2 Cloud Storage can streamline your workflow today!

The post Backblaze’s Must See List for NAB 2019 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

A Workflow Playbook for Migrating Your Media Assets to a MAM

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/workflow-playbook-migrating-your-media-assets-to-a-mam/

Asset > Metadata > Database > Media Asset Manager > Backblaze Fireball > Backblaze B2 Cloud Storage

This is one in a series of posts on professional media management leading up to NAB 2019 in Las Vegas, April 8 to 11.
–Editor

Whatever your creative venture, the byproduct of all your creative effort is assets. Whether you produce music, images, or video, as you produce more and more of these valuable assets, they tend to pile up and become difficult to manage, organize, and protect. As your creative practice evolves to meet new demands, and the scale of your business grows, you’ll often find that your current way of organizing and retrieving assets can’t keep up with the pace of your production.

For example, if you’ve been managing files by placing them in carefully named folders, getting those assets into a media asset management system will make them far easier to navigate and much easier to pull out exactly the media you need for a new project. Your team will be more efficient and you can deliver your finished content faster.

As we’ve covered before, putting your assets in a type of storage like B2 Cloud Storage ensures that they will be protected in a highly durable and highly available way that lets your entire team be productive.

You can learn about some of the new capabilities of the latest cloud-based collaboration tools here:

With some smart planning, and a little bit of knowledge, you can be prepared to get the most of your assets as you move them into an asset management system, or when migrating from an older or less capable system into a new one.

Assets and Metadata

Before we can build some playbooks to get the most from your creative assets, let’s review a few key concepts.

Asset — a rich media file with intrinsic metadata.

An asset is simply a file that is the result of your creative operation, and most often a rich media file like an image or a video. Typically, these files are captured or created in a raw state, then your creative team adds value to that raw asset by editing it together with other assets to create a finished story that in turn, becomes another asset to manage.

Metadata — Information about a file, either embedded within the file itself or associated with the file by another system, typically a media asset management (MAM) application.

The file carries information about itself that can be understood by your laptop or workstation’s operating system. Some of these seem obvious, like the name of the file, how much storage space it occupies, when it was first created, and when it was last modified. These would all be helpful ways to try to find one particular file you are looking for among thousands just using the tools available in your OS’s file manager.

File Metadata

There’s usually another level of metadata embedded in media files that is not so obvious but potentially enormously useful: metadata embedded in the file when it’s created by a camera, film scanner, or output by a program.

Results of a file inspected by an operating system's file manager
An example of metadata embedded in a rich media file

For example, this image taken in Backblaze’s data center a few years ago carries all kinds of interesting information. For example, when I inspect the file on macOS’s Finder with Get Info, a wealth of information is revealed. I can now not only tell the image’s dimensions and when the image was taken, but also exactly what kind of camera took this picture and the lens settings that were used, as well.

As you can see, this metadata could be very useful if you want to find all images taken on that day, or even images taken with that same camera, focal length, F-stop, or exposure.

When a File and Folder System Can’t Keep Up

Inspecting files one at a time is useful, but a very slow way to determine if a file is the one you need for a new project. Yet many creative environments that don’t have a formal asset management system get by with an ad hoc system of file and folder structures, often kept on the same storage used for production or even on an external hard drive.

Teams quickly outgrow that system when they find that their work spills over to multiple hard drives, or takes up too much space on their production storage. Worst of all, assets kept on a single hard drive are vulnerable to disk damage, or to being accidentally copied or overwritten.

Why Your Assets Need to be Managed

To meet this challenge, creative teams have often turned to a class of application called a Media Asset Manager (MAM). A MAM automatically extracts all their assets’ inherent metadata, helps move files to protected storage, and makes them instantly available to their entire team. In a way, these media asset managers become a private media search engine where any file attribute can be a search query to instantly uncover the file they need in even the largest media asset libraries.

Beyond that, asset management systems are rapidly becoming highly effective collaboration and workflow tools. For example, tagging a series of files as Field Interviews — April 2019, or flagging an edited piece of content as HOLD — do not show customer can be very useful indeed.

The Inner Workings of a Media Asset Manager

When you add files into an asset management system, the application inspects each file, extracting every available bit of information about the file, noting the file’s location on storage, and often creating a smaller stand-in or proxy version of the file that is easier to present to users.

To keep track of this information, asset manager applications employ a database and keep information about your files in it. This way, when you’re searching for a particular set of files among your entire asset library, you can simply make a query of your asset manager’s database in an instant rather than rifling through your entire asset library storage system. The application takes the results of that database query and retrieves the files you need.

The Asset Migration Playbook

Whether you need to move from a file and folder based system to a new asset manager, or have been using an older system and want to move to a new one without losing all of the metadata that you have painstakingly developed, a sound playbook for migrating your assets can help guide you.

Play 1 — Getting Assets in Files and Folders Protected Without an Asset Management System

In this scenario, your assets are in a set of files and folders, and you aren’t ready to implement your asset management system yet.

The first consideration is for the safety of the assets. Files on a single hard drive are vulnerable, so if you are not ready to choose an asset manager your first priority should be to get those files into a secure cloud storage service like Backblaze B2.

We invite you to read our post: How Backup and Archive are Different for Professional Media Workflows

Then, when you have chosen an asset management system, you can simply point the system at your cloud-based asset storage to extract the metadata of the files and populate the asset information in your asset manager.

  1. Get assets archived or moved to cloud storage
  2. Choose your asset management system
  3. Ingest assets directly from your cloud storage

Play 2 — Getting Assets in Files and Folders into Your Asset Management System Backed by Cloud Storage

In this scenario, you’ve chosen your asset management system, and need to get your local assets in files and folders ingested and protected in the most efficient way possible.

You’ll ingest all of your files into your asset manager from local storage, then archive them to cloud storage. Once your asset manager has been configured with your cloud storage credentials, it can automatically move a copy of local files to the cloud for you. Later, when you have confirmed that the file has been copied to the cloud, you can safely delete the local copy.

  1. Ingest assets from local storage directly into your asset manager system
  2. From within your asset manager system archive a copy of files to your cloud storage
  3. Once safely archived, the local copy can be deleted

Play 3 — Getting a Lot of Assets on Local Storage into Your Asset Management System Backed by Cloud Storage

If you have a lot of content, more than say, 20 terabytes, you will want to use a rapid ingest service similar to Backblaze’s Fireball system. You copy the files to Fireball, Backblaze puts them directly into your asset management bucket, and the asset manager is then updated with the file’s new location in your Backblaze B2 account.

This can be a manual process, or can be done with scripting to make the process faster.

You can read about one such migration using this play here:
iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted

  1. Ingest assets from local storage directly into your asset manager system
  2. Archive your local assets to Fireball (up to 70 TB at a time)
  3. Once the files have been uploaded by Backblaze, relink the new location of the cloud copy in your asset management system

You can read more about Backblaze Fireball on our website.

Play 4 — Moving from One Asset Manager System to a New One Without Losing Metadata

In this scenario you have an existing asset management system and need to move to a new one as efficiently as possible to not only take advantage of your new system’s features and get files protected in cloud storage, but also to do it in a way that does not impact your existing production.

Some asset management systems will allow you to export the database contents in a format that can be imported by a new system. Some older systems may not have that luxury and will require the expertise of a database expert to manually extract the metadata. Either way, you can expect to need to map the fields from the old system to the fields in the new system.

Making a copy of old database is a must. Don’t work on the primary copy, and be sure to conduct tests on small groups of files as you’re migrating from the older system to the new. You need to ensure that the metadata is correct in the new system, with special attention that the actual file location is mapped properly. It’s wise to keep the old system up and running for a while before completely phasing it out.

  1. Export the database from the old system
  2. Import the records into the new system
  3. Ensure that the metadata is correct in the new system and file locations are working properly
  4. Make archive copies of your files to cloud storage
  5. Once the new system has been running through a few production cycles, it’s safe to power down the old system

Play 5 — Moving Quickly from an Asset Manager System on Local Storage to a Cloud-based System

In this variation of Play 4, you can move content to object storage with a rapid ingest service like Backblaze Fireball at the same time that you migrate to a cloud-based system. This step will benefit from scripting to create records in your new system with all of your metadata, then relink with the actual file location in your cloud storage all in one pass.

You should test that your asset management system can recognize a file already in the system without creating a duplicate copy of the file. This is done differently by each asset management system.

  1. Export the database from the old system
  2. Import the records into the new system while creating placeholder records with the metadata only
  3. Archive your local assets to Fireball (up to 70 TB at a time)
  4. Once the files have been uploaded by Backblaze, relink the cloud based location to the asset record

Wrapping Up

Every production environment is different, but we all need the same thing: to be able to find and organize our content so that we can be more productive and rest easy knowing that our content is protected.

These plays will help you take that step and be ready for any future production challenges and opportunities.

If you’d like more information about media asset manager migration, join us for our webinar on March 15, 2019:

Backblaze Webinar:  Evolving for Intelligence: MAM to MAM Migration

•  •  •

Backblaze will be exhibiting at NAB 2019 in Las Vegas on April 8-11, 2019.NABShow logoSchedule a meeting with our cloud storage experts to learn how B2 Cloud Storage can streamline your workflow today!

The post A Workflow Playbook for Migrating Your Media Assets to a MAM appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Cloud-based Tools Combined with AI Can Make Workflows More Powerful and Increase Content Value

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/increase-content-archive-value-via-cloud-tools/

CPU + Metadata Mining + Virtual Machines & Apps + AI in the cloud

This is part two of a series. You can read part one at Modern Storage Workflows in the Age of Cloud.

Modern Storage Workflows in the Age of Cloud, Part 2

In Modern Storage Workflows in the Age of Cloud, Part One, we introduced a powerful maxim to guide content creators (anyone involved in video or rich media production) in choosing storage for the different parts of their content creation workflows:

Choose the storage that best fits each workflow step.

It’s true that every video production environment is different, with different needs, and the ideal solution for an independent studio of a few people is different than the solution for a 50-seat post-production house. But the goal of everyone in the business of creative storytelling is to tell stories and let your vision and craft shine through. Anything that makes that job more complicated and more frustrating keeps you from doing your best work.

Given how prevalent, useful, and inexpensive cloud technologies are, almost every team today is rapidly finding they can jettison whole classes of storage that are complicating their workflow and instead focus on two main types of storage:

  1. Fast, shared production storage to support editing for content creation teams (with no need to oversize or overspend)
  2. Active, durable, and inexpensive cloud storage that lets you move all of your content in one protected, accessible place — your cloud-enabled content backplane

It turns out there’s another benefit unlocked when your content backplane is cloud enabled, and it’s closely tied to another production maxim:

Organizing content in a single, well managed repository makes that content more valuable as you use it.

When all content is in a single place, well-managed and accessible, content gets discovered faster and used more. Over time it will pick up more metadata, with sharper and more refined tags. A richer context is built around the tags, making it more likely that the content you already have will get repurposed for new projects.

Later, when you come across a large content repository to acquire, or contemplate a digitization or preservation project, you know you can bring it into the same content management system you’ve already refined, concentrating and increasing value further still.

Having more content that grows increasingly valuable over time becomes a monetization engine for licensing, content personalization, and OTT delivery.

You might think that these benefits already present a myriad of new possibilities, but cloud technologies are ready to accelerate the benefits even further.

Cloud Benefits — Pay as You Need It, Scalability, and Burstability

It’s worth recapping the familiar cost-based benefits of the cloud: 1) pay only for the resources you actually use, and only as long as you need them, and, 2) let the provider shoulder the expense of infrastructure support, maintenance, and continuous improvement of the service.

The cost savings from the cloud are obvious, but the scalability and flexibility of the cloud should be weighted strongly when comparing using the cloud versus handling infrastructure yourself. If you were responsible for a large server and storage system, how would you cope with a business doubling every quarter, or merging with another team for a big project?

Too many production houses end up disrupting their production workflow (and their revenue) when they are forced to beef up servers and storage capability to meet new production demands. Cloud computing and cloud storage offer a better solution. It’s possible to instantly bring on new capacity and capability, even when the need is unexpected.

Cloud Delivered Compute Horsepower on Demand

Let’s consider the example of a common task like transcoding content and embedding a watermark. You need to process 3,600 frames of a two hour movie to resize the frame and add a watermark, and that compute workload takes 100 minutes and ties up a single server.

You could adapt that workflow to the cloud by pulling high resolution frames from cloud storage, feed them to 10 cloud servers in parallel, and complete the same job in 10 minutes. Another option is to spin up 100 servers and get the job done in one minute.

The cloud provides the flexibility to cut workflow steps that used to take hours down to minutes by adding the compute horsepower that’s needed for the job, then turn it off when it’s no longer needed. You don’t need to worry about planning ahead or paying for ongoing maintenance. In short, compute adapts to your workflow rather than the other way around, which empowers you to make workflow choices that instead prioritize the creative need.

Your Workflow Applications Are Moving to the Cloud, Too

More and more of the applications used for content creation and management are moving to the cloud, as well. Modern web browsers are gaining astonishing new capabilities and there is less need for dedicated application servers accompanying storage.

What’s important is that the application helps you in the creative process, not the mechanics of how the application is served. Increasingly, this functionality is delivered by virtual machines that can be spun up by the thousands as needed or by cloud applications that are customized for each customer’s specific needs.

iconik media workflow management screenshot

An example of a cloud-delivered workflow application — iconik asset discovery and project collaboration

iconik is one example of such a service. iconik delivers cloud-based asset management and project collaboration as a service. Instead of dedicated servers and storage in your data center, each customer has their own unique installation of iconik’s service that’s ready in minutes from first signup. The installation is exclusive to your organization and tailored to your needs. The result is a workflow utilizing virtual machines, compute, and storage that matches your workflow with just the resources you need. The resources are instantly available whenever or wherever your team is using the system, and consume no compute or storage resources when they are not.

Here’s an example. A video file can be pumped from Backblaze B2 to the iconik application running on a cloud compute instance. The proxies and asset metadata are stored in one place and available to every user. This approach is scalable to as many assets and productions you can throw at it, or as many people as are collaborating on the project.

The service is continuously upgraded and updated with new features and improvements as they become available, without the delay of rolling out enhancements and patches to different customers and locations.

Given the advantages of the cloud, we can expect that more steps in the creative production workflow that currently rely on dedicated on-site servers will move to the highly agile and adaptable environment offered by the cloud.

The Next Evolution — AI Becomes Content-Aware

Having your content library in a single content backplane in the cloud provides another benefit: ready access to a host of artificial intelligence (AI) tools.

Examples of AI Tools That Can Improve Creative Production Workflows:

  • Text to speech transcription
  • Language translation
  • Object recognition and tagging
  • Celebrity recognition
  • Brand use recognition
  • Colorization
  • High resolution conversion
  • Image stabilization
  • Sound correction

AI tools can be viewed as compute workers that develop processing rules by training for a desired result on a data set. An AI tool can be trained by having it process millions of images until it can tell the difference between sky and grass, or pick out a car in a frame of video. Once such a tool has been trained, it provides an inexpensive way to add valuable metadata to content, letting you find, for example, every video clip across your entire library that has sky, or grass, or a car in it. Text keywords with an associated timecode can be automatically added to aid in quickly zeroing in on a specific section of a long video clip. That’s something that’s not practical for a human content technician over thousands of files, but is easy, repeatable, and scalable for an AI tool.

Let AI Breathe New Life into Existing Content

AI tools can breathe new life in older content and intelligently clean up older format source video by removing film scratches or upresing content to today’s higher resolution formats. They can be valuable for digital restoration and preservation projects, too. With AI tools and source content in the cloud, it’s now possible to give new life to analog source footage. Digitize it, let AI clean it up, and you’ll get fresh, monetizable assets in your library.

axle ai automatically tags:

An example of the time-synched tags that can be generated with an AI tool

Many workflow tools, such as asset and collaboration tools, can use AI tools for speech transcription or smart object recognition, which brings additional capabilities. axle.ai, for example, can connect with a visual search tool to highlight an object in the frame like a wine bottle, letting you subsequently find every shot of a wine bottle across your entire library.

Visual search for brands and products also is possible. Just highlight a brand logo and find every clip where the camera panned over that logo. It’s smart enough to gets results even when only part of the logo is shown.

We’ve barely touched on the many tools that can be applied to content on ingest or content already in place. Whichever way they’re applied, they can deliver on the promise of making your workflows more efficient and powerful, and your content more valuable.

All Together Now

Taken together, these trends are great news for creatives. They can serve your creative vision by making your workflow more agile and more efficient. Cloud-enabled technologies enable you to focus on adding value and repurposing content in fresh new ways, resulting in new audiences and better monetization.

By placing your content in a cloud content backplane, and taking advantage of applications as a service, including the latest AI tools, it becomes possible to continually grow your content collection while increasing its value — a desirable outcome for any creative production enterprise.

If you could focus only on delivering great creative content, and had a host of AI tools to automatically make your content more valuable, what would you do?

The post Cloud-based Tools Combined with AI Can Make Workflows More Powerful and Increase Content Value appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Modern Storage Workflows in the Age of Cloud

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/cloud-based-video-production-workflows/

Video Production Workflow

Not too long ago, hardware storage vendors held an iron grip on what kinds of storage underpinned your creative, film, and broadcast workflows. This storage took many complex forms — RAIDs, JBODs, SANs, NAS systems, tape robots, and more. All of it was expensive, deeply complex, and carried fat vendor margins and high support costs.

How Storage Can Make Your Video Production Workflow More Efficient

But when you’re considering storage in today’s technology environment — whether it’s cloud, on-site storage, or a USB stick — the guiding principle in choosing storage for your creative production should simply be to choose the storage that best fits each workflow step.

Production Storage Maxim: Choose the storage that best fits each workflow step

Doing your best creative work is what builds your customer base, boosts your reputation, and earns you revenue and royalties. So any time sunk into legacy storage solutions, wrestling with complexity, unneeded production steps, refereeing competing vendors, and overpaying for, well, everything, just gets in the way of what you really want to do, create.

The right answer for your specific production needs is a function of the size of your production team and the complexity of your operating environment. Whatever that answer is, it should be as frictionless an environment as possible that helps you get your work done more efficiently and gives you the most flexibility.

An independent filmmaker can follow this production storage evaluation process for each stage of their workflow and decide to make do with a small deskside RAID system for primary production storage, and depend on the cloud for everything else.

A large, global production team will probably need multiple SANs in each production office and a complex series of cloud and dedicated playout applications and systems. If your environment falls somewhere between those two extremes, then your ideal solution mix does as well.

Traditional Content Production Workflow - Ingest > Work-in-Process > Deliver > Archive

The traditional content production workflow is thought of as a linear process. Content is ingested as raw camera files pulled into a shared work-in-process storage for editors, the final cut is then delivered to the client, and when the project is finished all files are saved off to an archive.

Simplified Production Workflow Steps

Let’s look at what the storage requirements and needs are for each of the common steps in a production workflow and where cloud can add value. Along the way, we’ll call out concrete examples of cloud capabilities at each stage with B2 cloud storage.

Ingest Stage - Ingest Stage Goals: Safely retrieve and protect files from capture media and move to production environment. Ingest Stage Needs: File data protection - Easy path to Production Storage. Where Cloud Can Add Value: Ingest and archive in one step

The Ingest Stage

Media copied in the ingest phase typically needs to get off of camera carts and flash drives as quickly and safely as possible and transported to the editing environment. Since those camera carts need to be used again for the next shot, pressure to get files copied over quickly (but safely) is intense.

Any time that critical content exists only in one place is dangerous. At this stage, lost or corrupted files mean a reshoot, which may not be practical or even possible.

Storage Needs for Ingest

Storage at the ingest stage can be very rudimentary and is often satisfied by just copying files from camera carts to an external drive, then to another drive as a safety, or by putting a RAID system on a crash cart on-set. Every team tends to come up with a different solution.

Where Cloud Can Add Value to Ingest

But even if your data wranglers aren’t ready to give up external hard drives here, one way cloud can help in the ingest stage is to help combine your ingest and archive for safety steps.

Instead of carrying carts from the shoot location to the production environment and copying them over to production storage, you could immediately start uploading content via the internet to your cloud storage, simultaneously copying over those files safely, and making them available to your entire team immediately.

When you restructure your workflow like this, you’ll get better than RAID-level protection for your content in the cloud. And by checking content into your archive first, your asset manager tools can immediately start processing those files by adding tags and generating lighter weight proxies. As soon as the files hit cloud storage, your entire team can start working on them. They can immediately begin tagging and reviewing files, and even mark edit points before handing off to editors, thereby speeding up production dramatically.

Some creatives have hit a roadblock in trying to take advantage of the cloud. Data transfer has historically been gated by the available upload bandwidth at your given location, but our customers have solved this in some interesting ways.

Producers, editors, and reporters are finding that even cellular 4G internet connections make it feasible to immediately start uploading raw shots to their cloud storage. Others make it routine to stop off at a data center or affiliate with excellent upload speeds on their way in from the field.

Either way, even novice shooters and freelancers can safely get content into your system quickly in a system that can be as simple as an upload bucket in your B2 account and making sure that your media or project manager tools are configured to watch those upload points.

Cloud Capability Example — Use a Backblaze Fireball to Rapidly Ingest Content

Backblaze offers a Rapid Ingest Service to help get large amounts of your content into your Backblaze account quickly. Backblaze ships you a 70TB storage system that you connect to your network and copy content to. When the system is shipped back to Backblaze, it is quickly moved directly into your B2 account, dramatically reducing ingest times.

 

Cloud Capability Example — Share Files Directly From Cloud

Archive.zip file in B2

An example of navigating to a file-review bucket in the B2 web interface to copy the direct sharing link to send to a reviewer

In addition to the archive on ingest technique, many customers share files for approval review or dailies directly from their Backblaze B2 account’s web interface.

If your B2 bucket for finished files is public, you can get a direct share link from the Backblaze account management website and simply send that to your customer, thereby eliminating a copy step.

You can even snapshot a folder of your content in B2, and have Backblaze ship it directly to your customer.

Work in Process Stage - WIP Stage Goals: Support collaborative, simultaneous editing of source files to finished content. WIP Stage Needs: Performance to support shared, collaborative editing access for many users. Very large file support. Where Cloud Can Add Value: Keeping expensive primary production storage running efficiently.

The Work-In-Process Stage

Work-in-process or primary production storage is the main storage used to support collaborative editing and production of content. The bulk of what’s thought of as collaborative editing happens in this stage.

For simplicity we’re combining several steps under the umbrella of work-in-process such as craft editing, voiceover, sound, ADR, special effects, and even color grading and finish etc. under a far simpler work-in-process step.

As audio, color grading and SFX steps get more complex, they sometimes need to be broken out into separate, extremely high performance storage such as more exotic (and expensive) flash-based storage that then feeds the result back to WIP storage.

Work-in-Process Stage Storage Needs

Storage performance requirements in this stage are extremely hard to meet, demanding the ability to serve multiple editors, each pulling multiple, extremely large streams of video files as they edit raw shots into a complex, visual story. Meeting this requirement usually requires either equipment intensive SAN, or a NAS that scales to eye-watering size and price.

Many production environments have gotten in the habit of keeping older projects and media assets on the shared production environment alongside current production files, knowing that if those files are needed they can be retrieved quickly. But this also means that production storage fills up quickly, and it’s tempting to let more and more users not involved in primary production have access to those files as well, both of which can slow down production storage and creation of your content.

Having to make a rush purchase to expand or add to your SAN is not fun, especially in the middle of a project, so regularly moving any files not needed for current production to your content archive is a great strategy to keep your production storage as light and small as possible so that it can last over several seasons.

Where Cloud Can Add Value to Work-in-Process

By regularly moving content from your production storage you keep it light, fast, and simpler to manage. But that content still needs to be readily available. Cloud is an excellent choice here as content is both immediately available and stored on highly resilient object storage. In effect, you’re lightening the burden on your primary storage, and using cloud as an always ready, expanding store for all of your content. We’ll explore this concept more in the archive stage.

Deliver Stage - Deliver Stage Goals: Securely deliver finished files to upstream/downstream clients. Deliver Stage Needs: High reliability. Separation from primary production storage. Where Cloud Can Add Value: Share files directly and securely from cloud without copying.

The Deliver Stage

The deliver stage, where your finished work is handed off to your customer, varies depending on what type of creative you are. Broadcast customers will almost always need dedicated playout server appliances, and others will simply copy files to where they’re needed by downstream customers, or upstream to a parent organization for distribution. But, at some level, we all have to deliver our work when it’s done.

Deliver Stage Storage Needs

Files for delivery should be moved off of your primary production storage and delivered in a separate workflow available to dedicated workflow or playout tools. Whatever the workflow, this storage needs to be extremely reliable and available for your customers whenever it is needed.

Where Cloud Can Add Value to Deliver

Whether content delivery in your workflow is met by copying files to a playout server or giving a finished file to a customer, cloud can help cut down on the number of steps to get the content to its final destination while giving you extreme reliability.

Cloud Capability Example — Serve Time-Limited Links to Content

Many customers use the Backblaze B2 API to add expiration limits that can last from seconds to a week to shared links:

B2 command-line

An example of using the B2 command-line tool to generate time-expiring tokens for content sharing and delivery

If your team is comfortable writing scripts to automate your workflow, this can be a powerful way to directly share files simply and quickly with tools provided by Backblaze.

For more information see this B2 Article: Get Download Authorization

 

Cloud Capability Example — Move Content Directly to Your Delivery and Distribution Servers

Serving your content to a wide audience via your website, content channel, or app is an increasingly popular way to deliver content. And thanks to our recent Cloudflare agreement, you can now move content from your B2 storage over to Cloudflare’s content delivery network at zero transfer cost for your content application or website.For more information see this B2 article: How to Allow Cloudflare to Fetch Backblaze B2 Content

Archive Stage - Archive Stage Goals: Securely deliver finished files to upstream/downstream clients. Archive Stage Needs: High reliability. Separation from primary prodcution storage. Where Cloud Can Add Value: Serve as your content backplane across all workflow steps.

The Archive Stage

At last, we come to the archive stage of content creation, traditionally thought of as the end of the traditional content creation chain, the source of the most frustration for creatives, and the hardest storage to size properly.

Traditionally, when a project or season of a show is finished, all of the files used to create the content are moved off of expensive primary production storage and stored on separate, highly reliable storage in case they are needed again.

Archive Stage Storage Needs

Archive storage needs to be a safe repository for all of the content that you’ve created. It should scale well at a sustainable price, and make all archived content available immediately when requested by your users and workflow tools like asset managers.

Tape was often chosen to store these archive files because it was cheaper than disk-based storage and offered good reliability. But choosing tape required a large investment in specialized tape systems, tape media, and the associated support contracts and maintenance.

Tape based archiving strategies usually rely on compressing content as it’s written to tape to hit the advertised storage capacity of tape media. But video content is already stored in a compressed container, so compressing those files as they’re written and retrieved from tape offers no advantage and only slows the process down.

Here we find the chief drawback of tape based content archives for many customers: the time required to retrieve content from those tape systems. As the pace of production has increased, many customers find they can no longer wait for tape systems to return archive sets or unarchive files.

Where Cloud Can Add Value to Archive

The archive stage is where cloud has the most impact on your entire workflow. The benefits of cloud itself are familiar: the ability to scale up or down instantly as your needs change, paying only for the storage you actually use, extremely high object storage file reliability, and availability anywhere there is a network connection.

Modern Content Production Workflow - Ingest > Archive as a Cloud Content Backplane ><Work-In-Process

Creating The Cloud Content Backplane

Having all of your content immediately available to your production storage and your asset management systems is emerging as the killer feature of cloud for production environments. By adding cloud, your content production goes from a linear process to a highly active one where content can freely check in and out of all of your other workflow steps as you’re producing content.

By shifting your content archives to cloud like Backblaze B2, you are creating, in effect, a cloud content backplane that supports your entire content creation and delivery process with these new capabilities:

  • New productions now have access to every file you might possibly need without waiting, letting you explore more creative choices
  • A single, authoritative content repository backing all of your creative production lets you phase out other storage and the associated management headaches and expense
  • You can now serve and deliver files directly from your cloud-based content archive with no impact on production storage
  • Having content in a single place means that your workflow tools like asset managers work better. You can find files across your entire content store instantly, and even archive or move files from your production storage to your cloud content archive automatically

The content not needed on your work-in-process storage is both highly protected and immediately available wherever you need it. Your entire workflow can get much simpler with fewer steps, and you can phase out storage you no longer need on-site.

Above all, you’ll have fewer steps between you and creating great content, and you’ll be able to explore new creative options faster while shifting to a pay-as-you-use-it model for all of your content storage.

In part two, we’ll explore the ways your new cloud-delivered content archive backplane can dramatically improve how you create, deliver, and monetize content with other cloud-based technologies in the age of cloud.

The post Modern Storage Workflows in the Age of Cloud appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Move Even Your Largest Archives to B2 with Fireball and Archiware P5

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/archiware-p5-cloud-backup/

Archiware P5 and Fireball

Backblaze B2’s reliability, scalability, and affordable, “pay only for what you use” pricing means that it’s an increasingly popular storage option for all phases of content production, and that’s especially true for media archiving.

By shifting storage to B2, you can phase out hard-to-manage and expensive local backup storage and clear space on your primary storage. Having all of your content in a single place — and instantly available — can transform your production and keep you focused on the creative process.

Fireball Rapid Ingest to Speed Your First Migration to Backblaze B2

Once you sign up for Backblaze B2, one tool that can speed an initial content migration tremendously is Backblaze’s Fireball rapid ingest service. As part of the service, Backblaze ships you a 70TB storage system. You then copy over all the content that you want in B2 to the Fireball system: all at local network speeds. Once the system is shipped to Backblaze, it’s quickly moved to your B2 account, a process far faster than uploading those files over the internet.

Setting Up Your Media Archive

Since manually moving files to archive and backing up project folders can be very time-consuming, many customers choose software like Archiware P5 that can manage this automatically. In P5’s interface you can choose files to add to archive libraries, restore individual files to your local storage from B2, and even browse all of your archive content on B2 with thumbnail previews, and more.

However, many media and entertainment customers have terabytes and terabytes of content in “archive” — that is, project files and content not needed for a current production, but necessary to keep nearby, ready to pull into a new production.

They’d love to get that content into their Backblaze B2 account and then manage it with an archive, sync, backup solution like Archiware P5. But the challenge facing too many is how to get all these terabytes up to B2 through the existing bandwidth in the office. Once the large, initial archive is loaded, the incrementals aren’t a problem, but getting years of backlog pushed up efficiently is.

For anyone facing that challenge, we’re pleased to announce the Archiware P5 Fireball Integration. Our joint solution provides any customer with an easy way to get all of their archives loaded into their B2 account without having to worry about bandwidth bottlenecks.

Archiware P5 Fireball Integration

A backup and archive manager like Archiware P5 is a great way to get your workflow under control and automated while ensuring that your content is safely and reliably stored. By moving your archives offsite, you get the highest levels of data protection while keeping your data immediately available for use anytime, anywhere.

With the newest release, Archiware P5 can archive directly to Fireball at fast, local network speeds. Then, once your Fireball content has been uploaded to your Backblaze account, a few clicks are all that is needed to point Archiware at your Backblaze account as the new location of your archive.

Finally, you can clear out those closets of hard drives and tape sets!

Archiware P5 to B2 workflow

Archiware P5 can now archive directly to Fireball at local network speeds, which are then linked to their new locations in your B2 accounts. With a few clicks you can get your entire archive uploaded to the B2 cloud without suffering any downtime or bandwidth issues.

For detailed information about configuring Archiware to archive directly to Fireball:

For more information about Backblaze B2 Fireball Rapid Ingest Service:

Archiware on Synology and QNAP NAS Devices

Archiware, NAS and B2

Archiware P5 can also now run directly on several Synology, QNAP, and G-Tech NAS systems to archive and move content to your Backblaze B2 account over the internet

With their most recent releases Archiware now supports several NAS system devices from QNAP, Synology, and G-Tech as P5 clients or servers.

The P5 software is installed as an application from the NAS vendor’s app store and runs directly on the NAS system itself without having to install additional hardware.

This means that all of your offices or departments with these NAS systems can now fully participate in your sync, archive, and backup workflows, and each of them can archive off to your central Backblaze B2 account.

For more information:

Archiware plus Backblaze: A Complete Front-to-Back Media Solution

Archiware P5, Fireball, and Backblaze B2 are all important parts of a great backup, archive, and sync plan. By getting all of your content into archive and B2, you’ll know that it’s highly protected, instantly available for new production workflows, and also readily discoverable through thumbnail and search capability.

With the latest version of P5, you not only have your entire production and backup workflows managed, with Fireball you can get even the largest and hardest to move archive safely and quickly into B2, as well!

For more information about the P5 Software Suite: Archiware P5 Software Suite

And to order a Fireball as part of our Rapid Ingest Service, start here: Backblaze B2 Fireball


You might also be interested in reading our recent guest post written by Marc N. Batschkus of Archiware about how to save time, money, and gain peace of mind with an archive solution that combines Backblaze B2 and Archiware P5.

Creating a Media Archive Solution with Backblaze B2 and Archiware P5

 

The post Move Even Your Largest Archives to B2 with Fireball and Archiware P5 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/iconik-and-backblaze-cloud-production-solution/

Cantemo iconik Plus Backblaze B2 for Media Cloud Production

Cantemo iconik Plus Backblaze B2 for Media Cloud Production

Many of our customers are archiving media assets in Backblaze B2, from long-running television productions, media distributors, AR/VR video creators, corporate video producers, houses of worship, and many more.

They are emptying their closets of USB hard drives, clearing off RAID arrays, and migrating LTO tapes to cloud storage. B2 has been proven to be the least expensive storage for their media archives, while keeping the archives online and accessible. Gone are the days of Post-its, clipboards, and cryptic drive labels defining whether old video footage can be found or not. Migrating archives from one form of storage to another will no longer suck up weeks and weeks of time.

So now that their archives are limitless, secure, always active, and available, the next step is making them actionable.

Our customers have been asking us — how can I search across all of my archives? Can I preview clips before I download the hi-res master, or share portions of the archive with collaborators around the world? Why not use the latest AI tools to intelligently tag my footage with metadata?

To meet all of those needs and more, we are excited to announce that Cantemo’s iconik cloud media management service now officially supports Backblaze B2.

iconik — A Media Management Service

iconik is an affordable and simple-to-use media management service that can read a Backblaze B2 bucket full of media and make it actionable. Your media assets are findable, sortable with full previews, and ready to pull into a new project or even right into your editor, such as Adobe Premiere, instantly.

Cantemo iconik user interface

iconik — Cantemo’s new media management service with AI features to find, sort, and even suggest assets for your project across your entire library

As a true media management service, iconik’s pricing model is a pay-as-you-go service, transparently priced per-user, per month. There are no minimum purchases, no servers to buy, and no large licensing fees to pay. To use iconik, all your users need is a web browser.

iconik Pricing

To get an idea of what “priced-per-user” might look like, most organizations will need at least one administrative user ($89/month), standard users ($49/month) who can organize content, create workflows, and ingest new media, and browse-only users ($19/month), who can search and download what they need. There’s also a “share-only” level that has no monthly charge that lets you incorporate customer and reviewer comments. This should accommodate teams of all kinds and all sizes.

Best of all, iconik is intelligent about how it uses storage, and while iconik charges small consumption fees for proxy storage, bandwidth, etc., they have found that for customers that bring media from Backblaze B2 buckets, consumption charges should be less than 5% of the monthly bill for user licenses.

As part of their launch promotion, if you get started in October, Cantemo will give Backblaze customers a $300 getting started credit!

You can sign up and get started here using the offer code of BBB22018.

Everwell’s Experience with iconik and Backblaze

One of the first customers to adopt iconik with Backblaze is Everwell, a video production company. Everwell creates a constant stream of videos for medical professionals to show in their waiting rooms. Rather than continuously buying upgrades for their in-house asset management system and local storage, iconik allows Everwell to shift their production to the cloud for all of their users. Their new solution will allow Everwell to manage their growing library of videos as new content constantly comes online, and kick off longer form productions with full access to all the assets they need across a fast-moving team that can be anywhere their production takes them.

collage of Everwll video images

Everwell is a fast-growing medical content developer for healthcare givers

To speed up their deployment of iconik, Everwell started with Backblaze’s data ingestion service, Fireball. Everwell copied their content to Fireball, and once back in the Backblaze data center, the data from Fireball was quickly added directly to Everwell’s B2 buckets. iconik could immediately start ingesting the content in place and make it available to every user.

Learn more about Backblaze B2 Fireball

With iconik and Backblaze, Everwell dramatically simplified their workflow as well, collapsing several critical workflow steps into one. For example, by uploading source files to Backblaze B2 as soon as they’re shot, Everwell not only reduces the need to stage local production storage at every site, they ingest and archive in a single step. Every user can immediately start work on their part of the project.

“The ‘everyone in the same production building’ model didn’t work for us any longer as our content service grew, with more editors and producers checking in content from remote locations that our entire team needed to use immediately. With iconik and Backblaze, we have what feels like the modern cloud-delivered production tool we’ve always wanted.”

— Loren Goldfarb, COO, Everwell

See iconik in Action at NAB NYC October 17-18

NAB Show New York - Media In Action October 17-18 2018

Backblaze is at NAB New York. Meet us there!

We’re excited to bring you several chances to see iconik and Backblaze working together.

The first is the NAB New York show, held October 17-18 at the Javits Center. iconik will be shown by Professional Video Technology in Booth N1432, directly behind Backblaze, Booth N1333.

Have you signed up for NAB NY yet? You can still receive a free exhibits pass by entering Backblaze’s Guest Code NY8842.

And be sure to sign up to meet with the Backblaze team at NAB by signing up on our calendar.

Attend the iconik and B2 Webinar on November 20

Soon after NAB NY, Backblaze and iconik will host a webinar to demo the solution called “3 Steps to Making Your Cloud Media Archive ‘active’ With iconik and Backblaze B2.” The webinar will be presented on November 20 and available on demand after November 20. Be sure to sign up for that too!

3 Steps Demo with: iconik and Backblaze B2 Cloud Storage

Sign up for the iconik/B2 Webinar

Don’t Miss the iconik October Launch Promotion

The demand for creative content is growing exponentially, putting more demands on your creative team. With iconik and B2, you can make all of your media instantly accessible within your workflows while adopting a infinitely scalable, pay only for what you use, storage solution.

To take advantage of the iconik October launch promotion and receive $300 free credit with iconik, sign up using the BBB22018 code.

The post iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Moving Tape Content to Backblaze Fireball with Canister

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/moving-tape-content-to-cloud-storage/


Canister for Fireball: LTO tape to Backblaze B2 migration made 'drag and drop' easy
If you shoot video on the run and wrangle video from multiple sources, you know that reliably offloading files from your camera carts, storage cards, or pluggable SSDs can be a logistical challenge. All of your source files need to be copied over, verified, and backed up before you can begin the rest of your post-production work.  It’s arguably the most critical step in your post-production workflow.

Knowing how critical this step is, videographers and data wranglers alike have long relied on an app for Mac and Windows called Hedge to take charge of their file copy and verification needs.


Hedge source and target progress

Hedge for Mac and Windows — drag and drop source file copy and verify tool

With an intuitive drag and drop interface, Hedge makes it simple to select your cards, disks, or other sources, identify your destination drives, then copy and verify using a custom “Fast Lane” engine to speed transfers dramatically. You can log when copies were completed, and even back up to multiple destinations in the same action, including your local SAN, NAS, or Backblaze Fireball, then on to your Backblaze B2 cloud storage.

But How Do You “Data-Wrangle” Tape Content to the Cloud?

But what if you have content, backup sets, or massive media archives on LTO tape?

You may find yourself in one of these scenarios:

  • You may have “inherited” an older LTO tape system that is having a hard time keeping up with your daily workflow, and you aren’t ready to sign up for more capital expense and support contracts.
  • You may have valuable content “stuck” on tape that you can’t easily access and want it on cloud for content monetization workflows that would overwhelm your tape system.
  • Your existing tape based workflow is working fine for now, but you want to get all of that content into the cloud quickly to get ready for future growth and new customers with a solution similar to Hedge.

While many people decide to move tape workflows to cloud for simple economic reasons, having all of that content securely stored in the cloud means that the individual files and entire folders can be instantly pulled into workflows and directly shared from Backblaze B2 with no need for copying, moving, restoring, or waiting.

For more information about how Backblaze B2 can replace LTO solutions, including an LTO calculator:  Backblaze LTO Replacement Calculator

Whichever scenario fits your need, getting tape content into the cloud involves moving a lot of content at once, and in a perfect world it would be as easy to drag and drop that content from tape to Backblaze B2!

Meet Canister for Fireball

To meet this exact need the team that developed Hedge have created an “LTO tape content to Fireball” solution called Canister for Fireball.

Fireball is Backblaze’s solution to help you quickly get massive amounts of data into Backblaze B2 Cloud Storage. When you sign up for the service, Backblaze sends you a 70TB Fireball that is yours to use for 30 days. Simply attach it to your local network and copy content over to the device at the speed of your local network. You’re free to fill up and send in your Fireball device as many times as needed. When Backblaze receives your Fireball with your files, all of the content is ingested directly into Backblaze’s data centers and appears in your Backblaze B2 online storage.

Backblaze B2 Fireball Rapid Ingest Service

Canister for Fireball makes it incredibly easy to move your content and archives from your tape device to your Backblaze B2 Fireball. With an intuitive interface similar to Hedge, Canister copies over and verifies files read from your tapes.

Using Canister with B2

flow chart for moving data from tape to the cloudInsert LTO tapes in your tape system and Canister for Backblaze will move them to your Backblaze B2 Fireball for rapid ingest into your B2 Cloud Storage


Cannister to Fireball user interfaceSelect from any tape devices with LTO media…

Cannister data progression screenshot…and watch the files on the tape copy and verify to your Backblaze B2 Fireball

Here’s how the solution works:

Steps to Migrate Your LTO Content to the Cloud with Canister for Fireball

  1. Order a Fireball system: As part of the signup step you will choose a B2 bucket that you’d like your Fireball content moved to.
  2. Connect your Fireball system to your network, making sure that the workstation that connects to your tape device can also mount the storage volume presented by your Backblaze Fireball.
  3. Install Canister for Fireball on your Mac workstation.
  4. Connect your tape device. Any tape system that can read your tapes and mount them as an LTFS volume will work. Canister will automatically mount tapes inside the app for you.
  5. Launch Canister for Fireball. You can now select the tape device volume as your source, the Fireball as your target, and copy the files over to your Fireball.
  6. Repeat as needed until you have copied and verified all of your tapes securely to your Fireball. You can fill and send in your Fireball as many times as needed during your 30 day period. (And you can always extend your loaner period.)
LTFS or Linear Tape File System is an industry adopted way to make the contents of an entire tape cartridge available as if it were a single volume of files. Typically, the tape stores a list of the files and their location on that tape in the beginning, or header of the tape. When a tape is read into your tape device, that directory section is read in and the tape system then presents it to you as a volume of files and folders. Say you want to select an individual file from that LTFS volume to copy to your desktop. When you move that to your desktop, the tape spools out to wherever that file is stored, reads the entire stream of tape containing that file, then finally copies it to your desktop. It can be a very slow process indeed and why many people choose to store content in cloud storage like Backblaze B2 so that they get instant access to every file.

Now — Put Your LTO Tape Ingest Plan Into Action

If you have content on tape that needs to get into your Backblaze B2 storage, Canister for Fireball and a Backblaze B2 Fireball are the perfect solution.

Canister for Fireball can be licensed for 30 days of use for $99 and includes priority support. The full version is $199. If you decide to upgrade from the 30 Day license you’ll pay only the difference to the full version.

Get more information about Canister for Fireball

And of course, make sure that you’ve ordered your Fireball:

Order a Backblaze B2 Fireball

Now with your content and archives no longer “trapped” on tape, you can browse them in your asset manager, share links directly from Backblaze B2, and have your content ready to pull into new content creation workflows by your team located anywhere in the world.

The post Moving Tape Content to Backblaze Fireball with Canister appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Creating a Media Archive Solution with Backblaze B2 and Archiware P5

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/creating-a-media-archive-solution/

Backblaze B2 Cloud Storage + Archiware P5= 7 Ways to Save

B2 + P5 = 7 Ways to Save Time, Money and Gain Peace of Mind with an Archive Solution of Backblaze B2 and Archiware P5

by Dr. Marc M. Batschkus, Archiware

This week’s guest post comes to us from Marc M. Batschkus of Archiware, who is well-known to media and entertainment customers, and is a trusted authority and frequent speaker and writer on data backup and archiving.

— Editor

Archiving has been around almost forever.

Roman "Archivum"Roman “Archivum” where scrolls were stored for later reference.

The Romans used the word “archivum” for the building that stored scrolls no longer needed for daily work. Since then, files have replaced scrolls, but the process has stayed the same and so today, files that are no longer needed for daily production can be moved to an archive.

Backup and Archive

Backblaze and Archiware complement each other in accomplishing this and we’ll show you how to get the most from this solution. But before we look at the benefits of archiving, let’s take a step back and review the difference between backup and archive.

A backup of your production storage protects your media files by replicating the files to a secondary storage. This is a cyclical process, continually checking for changed and new files, and overwriting files after the specified retention time is reached.

Archiving, on the other hand is a data migration, moving files that are no longer needed for daily production to (long-term) storage, yet keeping them easily retrievable. This way, all completed productions are collected in one place and kept for later reference, compliance, and re-use.

Think of backup as a spare tire, archive as winter tiresThink of BACKUP as a spare tire, in case you need it, and ARCHIVE as a stored set of tires for different needs.

To use an analogy:

  • Think of backup as the spare tire in the trunk.
  • Think of archive as the winter tires in the garage.

Both are needed!

Editor’s note: For more insight on “backup vs archive” have a look at What’s the Diff: Backup vs Archive.

Building a Media Archive Solution with Archiware P5 and Backblaze B2

Now that the difference between backup and archive is clear, let’s have a look at what an archive can do to make your life easier.

Archiware archive catalog transfering to B2 cloud storageArchiware P5 can be your interface to locate and manage your files, with Backblaze B2 as your ready storage for all of those files

P5 Archive connects to Backblaze B2 and offers the interface for locating files.

B2 + P5 = 7 Ways to Save Time and Money and Gain Peace-of-Mind

  1. Free up expensive production storage
  2. Archive from macOS, Windows, and Linux
  3. Browse and search the archive catalog with thumbnails and proxies
  4. Re-use, re-purpose, reference and monetize files
  5. Customize the metadata schema to fit your needs and speed up search
  6. Reduce backup size and runtime by moving files from production storage
  7. Protect precious assets from local disaster and for the long-term (no further migration/upgrade needed)

Archive as Mini-MAM

The “Mini-MAM” features of Archiware P5 help you to browse and find files easier than ever. Browse the archive visually using the thumbnails and proxy clips in the archive catalog. Search for specific criteria or a combination of criteria such as location or description.

Since P5 Archive lets you easily expand and customize metadata fields and menus, you can build the individual metadata schema that works best for you.

Technical metadata (e.g. camera type, resolution, lens) can be automatically imported from the file header into the metadata fields of P5 archive using a script.

The archive becomes the file memory of the company saving time and energy because now there is only one place to browse and search for files.

Mini MAM screenshotArchiware as “Mini-MAM” —  thumbnails, proxies, even metadata all within Archiware P5

P5 offers maximum flexibility and supports all storage strategies, be it cloud, disk or tape and any combination of the above.

For more information on Archiving with Archiware: Archiving with Archiware P5. For macOS, P5 Archive offers integration with the Finder and Final Cut Pro X via the P5 Archive App. For more information on integrated archiving with Final Cut Pro X: macOS Finder and Final Cut Pro X Integrated Archiving

You can start building an archive immediately with Backblaze B2 cloud storage because it allows you to do this without any additional storage hardware and upfront investment.

Backblaze B2 is the Best of Cloud

  • ✓  Saves investment in storage hardware
  • ✓  Access from anywhere
  • ✓  Storage on demand
  • ✓  Perpetual storage – no migration or upgrade of hardware
  • ✓  Financially advantageous (OPEX vs CAPEX)
  • ✓  Best price in its category

Backblaze B2 offers flexible access so that the archive can be accessed from several physical locations with no storage hardware needing to be moved.

P5 Archive supports consumable files as archive format. This makes the single files accessible even if P5 Archive is not present at the other location. This opens up a whole new world of possibilities for collaborative workflows that were not possible before.

Save Money with OPEX vs CAPEX

CAPEX vs. OPEXCAPital EXpenditures are the money companies spend to purchase major physical goods that will be used for more than one year. Examples in our field are investments in hardware such as storage and servers.

OPerating EXpenses are the costs for a company to run its business operations on a daily basis. Examples are rent and monthly cost for cloud storage like B2.

By using Backblaze B2, companies can save CAPEX and instead have monthly payments only for the cloud storage they use, while also saving maintenance and migration cost. Furthermore, migrating files to B2 makes expansion of high performance and costly production storage unnecessary. Over time this alone will make the archive pay off.

Now that you know how to profit from archiving with Archiware P5 and Backblaze B2, let’s look at the steps to build the best archive for you.

Connecting B2 cloud storage screenshot

Backblaze B2 is already a built-in option in P5 and works with P5 Archive and P5 Backup.

For detailed setup and best practice see:

Cloud Storage Setup and Best Practice for Archiware

Steps in Planning a Media Archive

Depending on the size of the archive, the number of people working with and using it, and the number of files that are archived, planning might be extremely important. Thinking ahead and asking the right questions ensures that the archive later delivers the value that it was built for.

Including people that will configure, operate, and use the system guarantees a high level of acceptance and avoids blind spots in your planning.

  1. Define users: who administers, who uses and who archives?
  2. Decide and select: what goes into the archive, and when?
  3. Which metadata are needed to describe the data needed (what will be searched for)?
  4. Actual security: on what operating system, hardware, software, infrastructure, interfaces, network and medium will be archived?
  5. What security requirements should be fulfilled: off-site storage, duplication, storage duration, test cycles of media, generation migration, etc.
  6. Retrieval:
    • Who searches?
    • With what criteria?
    • Who is allowed to restore?
    • On what storage?
    • For what use?

Metadata is the key to the archive and enables complex searches for technical and descriptive criteria.

Naming Conventions or “What’s in a File Name?”

The most robust metadata you can have is the file name. It can travel through different operating systems and file systems. The file name is the only metadata that is available all the time. It is independent of any database, catalog, MAM system, application, or other mechanism that can keep or read metadata. With it, someone can instantly make sense of a file that gets isolated, left over, misplaced or transferred to another location. Building a solid and intelligent naming convention for media files is crucial. Consistency is key for metadata. Metadata is a solid foundation for the workflow, searching and sharing files with other parties. The filename is the starting point.

Wrapping Up

There is much more that can make a media archive extremely worthwhile and efficient. For further reading I’ve made this free eBook available for more tips on planning and implementation.

eBook:  Data Management, Backup and Archive for Media Professionals — How to Protect Valuable Video Data in All Stages of the Workflow by Marc M. Batschkus

Start looking into the benefits an archive can bring you today. There is a 30-day fully featured trial license for Archiware P5 that can be combined with the Backblaze B2 free trial storage.

Trial License:  About Archiware P5 and 30-Day Trial

And of course, if you’re not already a Backblaze B2 customer, sign up instantly at the link below.

B2 Cloud Storage:  Instant Signup

— Dr. Marc M. Batschkus, Archiware

The post Creating a Media Archive Solution with Backblaze B2 and Archiware P5 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.