Start spreading the news, we’re leaving next week. We’re going to be a part of it, NAB New York!
Did you know that NAB Show — the premiere technology expo for media & entertainment held every year in Las Vegas — also offers a fall show in New York City? They do. And yes, we’ll be there. We couldn’t turn down an event in the media capital of the nation, if not the world.
New York City is home to the Big Four broadcast networks, and is a global center for newspaper, book, and magazine publishing, the music industry, independent filmmaking, and online media. That makes New York a great place to visit existing customers and to meet new ones.
New York Customers That Backblaze
A number of New York-based creative agencies are B2 customers, including Baron & Baron, the agency that’s the driving force behind some of the world’s most celebrated luxury fashion and beauty brands. Our creative director and co-founder, Casey, got a big kick out of laying out their case study. It’s not every day that he gets to work with eye-catching imagery from the likes of Coach, Dior, and Armani. But that’s just one agency of many long-term B2 customers based in New York City. That’s not surprising since seven of the top eight global advertising agencies are headquartered in NYC. (Cue the theme music to Mad Men.)
New York City is also ground central for the growing number of online media companies, from online-only news outlets like Huffpost, Business Insider, Vox, and Vice, to digital-first brands and online instructional sites like B2 customer Panna Cooking. Being digital natives, it’s no surprise that online media companies have been quick to adopt cloud storage.
Finally, one-third of all American independent films are produced in New York City. (Don’t tell Hollywood!) At NAB New York we will be debuting a new case study for a documentary filmmaker whose craft has him riding the waves all over the world. Watching his films, you can almost feel the spray on your face and the waves beneath your feet.
Visit Backblaze at NAB New York
If you’ll be in New York on October 16-17, please stop by our booth #N756 at the Javits Center. We’ll be celebrating 10 years of our Backblaze Storage Pod, so we’ll have our latest pod in our booth for you to get up close and personal with one of Silicon Valley’s first open source hardware projects. If you want to spend more time exploring how B2 could fit into your workflow, our calendars are open for you to schedule an appointment and we’ll have a special gift waiting for you.
Our distributor PVT will also offer live demos across the aisle in their booth #N757, showing B2 in action with integrated workflow applications from Archiware and iconik. You can also visit other integrated application partners at their own booths, including CatDV #N660, EditShare #N345, Facilis #N136, FileCatalyst #N159, GB Labs #N359, and GrayMeta #N1120.
Finally, if you don’t have a pass to the show, register using our code NY9776 and the expo hall, on-floor sessions, and select networking events are completely FREE.
Only in New YorkTop Picks from Backblaze Staff
Of course, a trip to New York City isn’t all business. The Backblaze folks attending the show make a point of carving out time for some of their favorite things you can only experience in New York. And we’d like to share some of our faves with you.
Nilay Patel, Vice President of Sales When Nilay can squeeze it in, he heads to the Upper East Side for a real gentleman’s haircut at Paul Molé barber shop, New York City’s first. After that, he’ll be ready to stroll in style down the High Line park and up to the rooftop bar at the Standard Hotel.
Terry LoBianco, Director of Sales Terry feels inspired by the grand architecture of New York City, and Grand Central Terminal’s Beaux Arts design is among her favorites. Her plan is to stop in for cocktails at the Campbell Bar inside the station. She’ll be having a Manhattan of course!
Skip Levens, Director of Product Marketing I don’t know if it qualifies as traditional New York pizza, but Skip is salivating just remembering the pizza the team had at Tappo Thin Crust Pizza last year. Like a good native Californian, Skip craves the salads at Tappo too. And after he tears down our booth, he hopes to catch the opening of the Chelsea Film Festival that’s a hop, skip, and a jump from Javits.
Elton Carneiro, Director of Partnerships Poor Elton. He’s been craving Goa-style food at Bombay Bread Bar. But alas, celebrity Chef Floyd Cardoz closed up shop just two weeks before his arrival. He’s crossing his fingers that it will reopen somewhere else in the city. Anyone know? If not, Elton will have to wait for a trip to Mumbai.
Vincent Montez, Business Development Representative Vincent was quick to report that the Flat Iron Room was his top pick. His dream is to splurge for the pricey Ultimate Japanese whisky flight, but given his budget, he’ll probably settle on the more reasonably priced and geographically appropriate New York flight.
Pavithra Hari, Senior Sales Engineer Pavithra has it all planned. First she’ll hit the Halal Guys original food truck on 53rd & 6th, then take the subway to the Empire State Building for the long elevator ride to the top. She’s well aware that there’s a Halal Guys restaurant a few blocks from Backblaze headquarters, but it’s just not the same.
Patrick Thomas, Head of Publishing Leave it to Patrick to find the most healthy and introspective activities of the team: jogging on the Hudson River Greenway, sipping Japanese matcha tea at Ippodo, and if he’s lucky, catching an event at the New York City Poetry Society.
Janet Lafleur, Senior Product Marketing Manager Sadly, Janet won’t be joining the crew in New York this year — someone has to hold down the fort. But she already has a plan for NAB New York 2020: jazz and soul food at Red Rooster in Harlem. Red Rooster comes highly recommended by Shaneika, a Backblaze account executive and native New Yorker. The seafood jambalaya will have to wait.
Prost!Skål!Cheers! Celebrate with us as we travel to Amsterdam for IBC, the premier conference and expo for media and entertainment technology in Europe. The show gives us a chance to raise a glass with our partners, customers, and future customers across the pond. And we’re especially pleased that IBC coincides with the opening of our new European data center.
How will we celebrate? With the Backblaze Partner Crawl, a rolling series of parties on the show floor from 13-16 September. Four of our Europe-based integration partners have graciously invited us to co-host drinks and bites in their stands throughout the show.
If you can make the trip to IBC, you’re invited to toast us with a skål! with our Swedish friends at Cantemo on Friday, a prost! with our German friends at Archiware on Saturday, or a cheers! with UK-based friends at Ortana and GB Labs on Sunday or Monday, respectively. Or drop in every day and keep the Backblaze Partner Crawl rolling. And if you can’t make it to IBC this time, we encourage you to raise a glass and toast anyway.
Skål! on Friday With Cantemo
Cantemo’s iconik media management makes sharing and collaborating on media effortless, regardless of wherever you want to do business. Cantemo announced the integration of iconik with Backblaze’s B2 Cloud Storage last fall, and since then we’ve been amazed by customers like Everwell, who replaced all their on-premises storage with a fully cloud-based production workflow. For existing Backblaze customers, iconik can speed up your deployment by ingesting content already uploaded to B2 without having to download files and upload them again. You can also stop by the Cantemo booth anytime during IBC to see a live demo of iconik and Backblaze in action. Or schedule an appointment and we’ll have a special gift waiting for you.
Join us at Cantemo on Friday 13 September from 16:30-18:00 at Hall 7 — 7.D67
Prost! on Saturday With Archiware
With the latest release of their P5 Archive featuring B2 support, Archiware makes archiving to the cloud even easier. Archiware customers with large existing archives can use the Backblaze Fireball to rapidly import archived content directly to their B2 account. At IBC, we’re also unveiling our latest joint customer, Baron & Baron, a creative agency that turned to P2 and B2 to back up and archive their dazzling array of fashion and luxury brand content.
Join us at Archiware on Saturday 14 September from 16:30-18:00 at Hall 7 — 7.D35
Cheers! on Sunday With Ortana
Ortana integrated their Cubix media asset management and orchestration platform with B2 way back in 2016 during B2’s beta period, making them among our first media workflow partners. More recently, Ortana joined our Migrate or Diewebinar and blog series, detailing strategies for how you can migrate archived content from legacy platforms before they go extinct.
Join us at Ortana on Sunday 15 September from 16:30-18:00 at Hall 7 — 7.C63
Cheers! on Monday With GB Labs
If you were at the NAB Show last April, you may have heard GB Labs was integrating their automation tools with B2. It’s official now, as detailed in their announcement in June. GB Labs’ automation allows you to streamline tasks that would otherwise require tedious and repetitive manual processes, and now supports moving files to and from your B2 account.
Join us at GB Labs Monday 16 September from 17:00-18:00 at Hall 7 — 7.B26
Say Hello Anytime to Our Friends at CatDV
CatDV media asset management helps teams organize, communicate, and collaborate effectively, including archiving content to B2. CatDV has been integrated with B2 for over two years, allowing us to serve customers like UC Silicon Valley, who built an end-to-end collaborative workflow for a 22 member team creating online learning videos.
Stop by CatDV anytime at Hall 7 — 7.A51
But we’re not the only ones making a long trek to Amsterdam for IBC. While you’re roaming around Hall 7, be sure to stop by our other partners traveling from near and far to learn what our joint solutions can do for you:
EditShare (shared storage with MAM) Hall 7 — 7.A35
ProMax (shared storage with MAM) Hall 7 — 7.D55
StorageDNA (smart migration and storage) Hall 7 — 7.A32
FileCatalyst (large file transfer) Hall 7 — 7.D18
eMAM (web-based DAM) Hall 7 — 7.D27
Facilis Technology (shared storage) Hall 7 — 7.B48
GrayMeta (metadata extraction and insight) Hall 7 — 7.D25
Hedge (backup software) Hall 7 — 7.A56
axle ai (asset management) Hall 7 — 7.D33
Tiger Technology (tiered data management) Hall 7 — 7.B58
We’re hoping you’ll join us for one or more of our Partner Crawl parties. If you want a quieter place and time to discuss how B2 can streamline your workflow, please schedule an appointment with us so we can give you the attention you need.
Finally, if you can’t join us in Amsterdam, open a beer, pour a glass of wine or other drink, and toast to our new European data center, wherever you are, in whatever language you speak. As we say here in the States, Bottoms up!
I first met Laura D’Antoni when we were shooting B2 Cloud Storage customer videos for Youngevity and Austin City Limits. I enjoyed talking about her filmmaking background and was fascinated by her journey as a director, editor, and all around filmmaker. When she came to the Backblaze office to shoot our Who We Are and What We Do video, I floated the idea of doing an interview with her to highlight her journey and educate our blog readers who may be starting out or are already established in the filmmaking world. We’ve finally gotten around to doing the interview, and I hope you enjoy the Q&A with Laura below!
Q: How did you get involved in visual storytelling? My interest in directing films began when I was 10 years old. Back then I used my father’s Hi8 camera to make short films in my backyard using my friends as actors. My passion for filmmaking continued through my teens and I ended up studying film and television at New York University.
Q: Do you have a specialty or favorite subject area for your films? I’ve always been drawn to dramatic films, especially those based on real life events. My latest short is a glimpse into a difficult time in my childhood, told in reverse Memento-style from a little girl’s perspective.
Most of my filmmaking career I actually spent in the documentary world. I’ve directed a few feature documentaries about social justice and many more short docs for non-profit organizations like the SPCA.
Q: Who are you visual storyteller inspirations? What motivates you to tell your stories? The film that inspired me the most when I was just starting out was The Godfather: Part II. The visuals and the performances are incredible, and probably my father being from Sicily really drew me in (the culture, not the Mafia, ha!). Lately I’ve been fascinated by the look of The Handmaid’s Tale, and tried to create a similar feel for my film on a much, much tinier budget. As far as what motivates me, it’s the love for directing. Collaborating with a team to make your vision on paper a reality is an incredible feeling. It’s a ton of work that involves a lot of blood, sweat, and tears, but in the end you’ve made a movie! And that’s pretty cool.
Q: What kind of equipment do you take on shoots? Favorite camera, favorite lens? For shoots I bring lights, cameras, tripods, a slider and my gimbal. I use my Panasonic EVA-1 as my main camera and also just purchased the Panasonic GH5 as B-cam to match. Most of my lenses are Canon photo lenses; the L-glass is fantastic quality and I like the look of them. My favorite lens is the Canon 70-200mm f2.8.
Q: How much data per day does a typical shoot create? If I’m shooting in 4K, around 150GB.
Q: How do you back up your daily shoots? Copy to a disk? Bunch of disks? I bring a portable hard drive and transfer all of the footage from the cards to that drive.
Q: Tell us a bit about your workflow from shooting to editing. Generally, if the whole project fits onto a drive, I’ll use that drive to transfer the footage and then edit from it as well. If I’ve shot in 4K then the first step before editing is creating proxies in Adobe Premiere Pro of all of the video files so it’s not so taxing on my computer. Once that’s done I can start the edit!
Q: How do you maintain your data? If it’s a personal project, I have two copies of everything on separate hard drives. For clients, they usually have a backup of the footage on a drive at their office. The data doesn’t really get maintained, it just stays on the drive and may or may not get used again.
Q: What are some best practices for keeping track of all your videos and assets? I think having a Google Docs spreadsheet and numbering your drives is helpful so you know what footage/project is where.
Q: How has having a good backup and archive strategy helped in your filmmaking? Well, I learned the hard way to always back up your footage. Years ago while editing a feature doc, I had an unfortunate incident with PluralEyes software and it ate the audio of one of my interview subjects. We ended up having to use the bad camera audio and nobody was happy. Now I know. I think the best possible strategy really is to have it backed up in the cloud. Hard drives fail, and if you didn’t back that drive up, you’re in trouble. I learned about a great cloud storage solution called Backblaze when I created a few videos for them. For the price it’s absolutely the best option and I plan on dusting off my ancient drives and getting them into the cloud, where they can rest safely until someday someone wants to watch a few of my very first black and white films!
Q: What advice do you have for filmmakers and videographers just starting out? Know what you want to specialize in early on so you can focus on just that instead of many different specialties, and then market yourself as just that.
It also seems that the easiest way into the film world (unless you’re related to Steven Spielberg or any other famous person in Hollywood) is to start from the bottom and work your way up.
Also, remember to always be nice to the people you work with, because in this industry that PA you worked with might be a big time producer before you know it.
Q: What might our readers find surprising about challenges you face in your work? In terms of my directing career, the most challenging thing is to simply be seen. There is so much competition, even among women directors, and getting your film in front of the right person that could bring your career to the next level is nearly impossible. Hollywood is all about who you know, not what you know, unfortunately. So I just keep on making my films and refuse to give up on my dream of winning an Academy Award for best director!
Q: How has your workflow changed since you started working with video? I only worked with film during my college years. It definitely teaches you to take your time and set up that shot perfectly before you hit record,; or triple check where you’re going to cut your film before it ends up on the floor and you have to crawl around and find it to splice it back in. Nowadays that’s all gone. A simple command- z shortcut and you can go back several edits on your timeline, or you can record countless hours on your video camera because you don’t have to pay to have it developed. My workflow is much easier, but I definitely miss the look of film.
Three examples of common media workflows using a NAS
Top five benefits of using NAS for photography and videography
The camera might be firmly entrenched at the top of the list of essential equipment for photographers and videographers, but a strong contender for next on the list has to be network-attached storage (NAS).
A big reason for the popularity of NAS is that it’s one device that can do so many things that are needed in a media management workflow. Most importantly, NAS systems offer storage larger than any single hard drive, let you centralize photo storage, protect your files with backups and data storage virtualization (e.g. RAID), allow you to access files from anywhere, integrate with many media editing apps, and securely share media with coworkers and clients. And that’s just the beginning of the wide range of capabilities of NAS. It’s not surprising that NAS has become a standard and powerful data management hub serving the media professional.
This post is an overview of how NAS can fit into the professional or serious amateur photo and video workflow and some of the benefits you can receive from adding a NAS.
Essential NAS Capabilities
Firstly, NAS is a data storage device. It connects to your computer, office, and the internet, and supports loading and retrieving data from multiple computers in both local and remote locations.
The number of drives available for data storage is determined by how many bays the NAS has. As larger and faster disk drives become available, a NAS can be upgraded with larger drives to increase capacity, or multiple NAS can be used together. Solid-state drives (SSDs) can be used in a NAS for primary storage or as a cache to speed up data access.
Data Protection and Redundancy
NAS can be used for either primary or secondary local data storage. Whichever it is, it’s important to have an off-site backup of that data, as well, to provide redundancy in case of accident, or in the event of a hardware or software problem. That off-site backup can be drives stored in another location, or more commonly these days, the cloud. The most popular NAS systems typically offer built-in tools to automatically sync files on your NAS to offsite cloud storage, and many also have app stores with backup and many other types of applications, as well.
Data is typically stored on the NAS using some form of error checking and virtual storage system, typically RAID 5 or RAID 6, to keep your data available even if one of the internal hard drives fail. However, if NAS is the only backup you have, and a drive fails, it can take quite a while to recover that data from a RAID device, and the delay only gets longer as drives increase in size. Avoiding this delay is the motivation for many to keep a redundant copy in the cloud so that it’s possible to access the files immediately even before the RAID has completed its recovery.
If your primary data files are on an editing workstation, the NAS can be your local backup to make sure you keep your originals safe from accidental changes or loss. In some common editing workflows, the raw files are stored on the NAS and lower-resolution, smaller proxies are used for offline editing on the workstation — also called non-destructive or non-linear editing. Once edits are completed, the changes are written back to the NAS. Some applications, including Lightroom, maintain a catalog of files that is separate from the working files and is stored on the editing workstation. This catalog should be routinely backed up locally and remotely to protect it, as well.
The data on the NAS also can be protected with automated data backups or snapshots that protect data in case of loss, or to retrieve an earlier version of a file. A particularly effective plan is to schedule off-hours backups to the cloud to complete the off-site component of the recommended 3-2-1 backup strategy.
Data Accessibility and Sharing
Data can be loaded onto the NAS directly through a USB or SD card slot, if available, or through any device available via the local network or internet. Another possibility is to have a directory/folder on a local computer that automatically syncs any files dropped there to the NAS.
Once on the NAS, files can be shared with coworkers, clients, family, and friends. The NAS can be accessed via the internet from anywhere, so you can easily share work in progress or final media presentations. Access can be configured by file, directory/folder, group, or by settings in the particular application you are using. NAS can be set up with a different user and permission structure than your computer(s), making it easy to grant access to particular folders, and keeping the security separate from however local computers are set up. With proper credentials, a wide range of mobile apps or a web browser can be used to access the data on the NAS.
Media Editing Integration
It’s common for those using applications such as Adobe Lightroom to keep the original media on the NAS and work on a proxy on the local computer. This speeds up the workflow and protects the original media files. Similarly, for video, some devices are fast enough to support NLE (non-linear editing), and therefore support using the NAS for source and production media but allow editing without changing the source files. Popular apps that support NLE include Adobe Premiere, Apple Final Cut Pro X, and Avid Media Composer.
Flexibility and Apps
NAS from Synology, QNAP, FreeNAS/TrueNAS, Morro Bay, and others offer a wide range of apps that extend the functionality of the device. You can easily turn a NAS into a media server that streams audio and video content to TVs and other devices on your network. You can set up a NAS to automatically perform backups of your computers, or configure that NAS as a file server, a web server, or even a telephone system. Some home offices and small businesses have even completely replaced office servers with NAS.
Examples of Common Media Workflows Using a NAS
The following are three examples of how a NAS device can fit into a media production workflow.
Example One — A Home Studio
NAS is a great choice for a home studio that needs additional data storage, file sharing, cloud backup, and secure remote access. NAS is a better choice than using directly-attached storage because it can have separate security than local computers and is accessible both locally and via the internet even when individual workstations might be turned off or disconnected.
NAS can provide centralized backup using common backup apps, including Time Machine and ChronoSync on Mac, or Backup and Restore and File History on Windows.
To back up to the cloud, major NAS providers, including Synology, QNAP, Morro Data, and FreeNAS/TrueNAS include apps that can automatically back up NAS data to B2 or other destinations on the schedule of your choice.
Example Two — A Distributed Media Company with Remote Staff
The connectivity of NAS makes it an ideal hub for a distributed business. It provides a central location for files that can be reliably protected with RAID, backups, and access security, yet available to any authorized staff person no matter where they are located. Professional presentations are easy to do with a range of apps and integrations available for NAS. Clients can be given controlled access to review drafts and final proofs, as well.
Example Three — Using NAS with Photo/Video Editing Applications
Many media pros have turned to NAS for storing their ever-growing photos and video data files. Frequently, these users will optimize their workstation for the editing or cataloging application of their choice using fast central and graphics processors, SSD drives, and large amounts of RAM, and offload the data files to the NAS.
While Adobe Lightroom requires that its catalog be kept on a local or attached drive, the working files can be stored elsewhere. Some users have adopted the digital negative (DNG) for working files, which avoids having to manage sidecar (XMP) files. XMP files are stored alongside the RAW files and record edits for file formats that don’t support saving that information natively, such as proprietary camera RAW files, including CRW, CR2, NEF, ORF, and so on.
With the right software and hardware, NAS also can play well in a shared video editing environment, enabling centralized storage of data with controlled access, file security, and supporting other functions such as video transcoding.
Top 5 Benefits of Using NAS for Photography and Videography
To recap, here are the top five benefits of adding NAS to your media workflow.
Flexible and expandable storage — fast, expandable and grows with your needs
Data protection — provides local file redundancy as well as an automated backup gateway to the cloud
Data accessibility and sharing — functions as a central media hub with internet connectivity and access control
Integration with media editing tools — works with editing and cataloging apps for photo and video
Flexibility and apps — NAS can perform many of the tasks once reserved for servers, with a wide range of apps to extend its capabilities
To learn more about what NAS can do for you, take a look at the posts on our blog on specific NAS devices from Synology, QNAP, FreeNAS/TrueNAS, and Morro Data, and about how to use NAS for photo and video storage. You’ll also find more information about how to connect NAS to the cloud. You can quickly find all posts on the NAS topic on our blog by following the NAS tag.
Do you have experience using NAS in a photo or video workflow? We’d love to hear about your experiences in the comments.
Apple’s annual WWDC is highlighting high-end desktop computing, but it’s laptop computers and the cloud that are driving a new wave of business and creative collaboration
WWDC, Apple’s annual megaconference for developers kicks off this week, and Backblaze has team members on the ground to bring home insights and developments. Yet while everyone is drooling over the powerful new Mac Pro, we know that the majority of business users use a portable computer as their primary system for business and creative use.
The Rise of the Mobile, Always On, Portable Workstation
After all, these systems are extremely popular with users and the DevOps and IT teams that support them. Small and self-contained, with massive compute power, modern laptops have fast SSD drives and always-connected Wi-Fi, helping users be productive anywhere: in the field, on business trips, and at home. Surprisingly, companies today can deploy massive fleets of these notebooks with extremely lean staff. At the inaugural MacDevOps conference a few years ago Google’s team shared that they managed 65,000 Macs with a team of seven admins!
<BOGGLE> Compare this to 99% of the clay feet ‘enterprise’ companies inflicting 90’s IT on their workforce. https://t.co/plHQSTRPLa
With the trend towards leaner IT staffs, and the dangers of computers in the field being lost, dropped or damaged, having a reliable backup system that just works is critical. Despite the proliferation of teams using shared cloud documents and email, all of the other files on your laptop you’re working on — the massive presentation due next week or the project that’s not quite ready to share on Google Drive — all have no protection without backup, which is of course why Backblaze exists!
Cloud as a Shared Business Content Hub is Changing Everything
When a company is backing up users’ files comfortably to the cloud, the next natural step is to adopt cloud-based storage like Backblaze B2 for your teams. With over 750 petabytes of customer data under management, Backblaze has worked with businesses of every size as they adopt cloud storage. Each customer and business does so for different reasons.
In the past, a business department typically would get a share of a company’s NAS server and was asked to keep all of the department’s shared documents there. But outside the corporate firewall, it turns out these systems are hard to access remotely from the road. They require VPNs and a constant network connection to mount a corporate shared drive via SMB or NFS. And, of course, running out of space and storing large files was an ever present problem.
Sharing Business Content in the Cloud Can be Transformational for Businesses
When considering a move to cloud-based storage for your team, some benefits seem obvious, but others are more profound and show that cloud storage is emerging as a powerful, organizing platform for team collaboration.
Shifting to cloud storage delivers these well-known benefits:
Pay only for storage you actually need
Grow as large and as quickly as you might need
Service, management, and upgrades are built in to the service
Pay for service as you use it out of operating expenses vs. onerous capital expenses
But shifting to shared, cloud storage yields even more profound benefits:
Your Business Content is Easier to Organize and Manage: When your team’s content is in one place, it’s easier to organize and manage, and users can finally let go of stashing content all over your organization or leaving it on their laptops. All of your tools to mine and uncover your business’s content work more efficiently, and your users do as well.
You Get Simple Workflow Management Tools for Free: Storage can fit your business processes much easier with cloud storage and do it on the fly. If you ever need to set up separate storage for teams of users, or define read/write rules for specific buckets of content, it’s easy to configure with cloud storage.
You Can Replace External File-Sharing Tools: Since most email services balk at sending large files, it’s common to use a file sharing service to share big files with other users on your team or outside your organization. Typically this means having to download a massive file, re-upload it to a file-sharing service, and publish that file-sharing link. When your files are already in cloud, sharing it is as simple as retrieving a URL location.
In fact, this is exactly how Backblaze organizes and serves PDF content on our website like customer case studies. When you click on a PDF link on the Backblaze website, it’s served directly from one of these links from a B2 bucket!
You Get Instant, Simple Policy Control over Your Business or Shared Content: B2 offers simple-to-use tools to keep every version of a file as it’s created, keep just the most recent version, or choose how many versions you require. Want to have your shared content links time-out after a day or so? This and more is all easily done from your B2 account page:
You’re One Step Away from Sharing That Content Globally: As you can see, beyond individual file-sharing, cloud storage like Backblaze B2 can serve as your origin store for your entire website. With the emergence of content delivery networks (CDN), you’re now only a step away from sharing and serving your content globally.
Get Sophisticated Content Discovery and Compliance Tools for Your Business Content: With more and more business content in cloud storage, finding the content you need quickly across millions of files, or surfacing content that needs special storage consideration (for GDPR or HIPAA compliance, for example) is critical.
Ideally, you could have your own private, customized search engine across all of your cloud content, and that’s exactly what a new class of solutions provide.
With Acembly or Aparavi on Backblaze, you can build content indexes and offer deep search across all of your content, and automatically apply policy rules for management and retention.
Where Are You in the Cloud Collaboration Trend?
The trend to mobile, always-on workers building and sharing ever more sophisticated content around cloud storage as a shared hub is only accelerating. Users love the freedom to create, collaborate and share content anywhere. Businesses love the benefits of having all of that content in an easily managed repository that makes their entire business more flexible and less expensive to operate.
So, while device manufacturers like Apple may announce exciting Pro level workstations, the need for companies and teams to collaborate and be effective on the move is an even more important and compelling issue than ever before. The cloud is an essential element of that trend that can’t be underestimated.
Like many Backblaze customers, Nodecraft realized they could save a fortune by shifting their cloud storage to Backblaze and invest it elsewhere in growing their business. In this post that originally appeared on Nodecraft’s blog, Gregory R. Sudderth, Nodecraft’s Senior DevOps Engineer, shares the steps they took to first analyze, test, and then move that storage. — Skip Levens
TL;DR: Nodecraft moved 23TB of customer backup files from AWS S3 to Backblaze B2 in just 7 hours.
Nodecraft.com is a multiplayer cloud platform, where gamers can rent and use our servers to build and share unique online multiplayer servers with their friends and/or the public. In the course of server owners running their game servers, there are backups generated including the servers’ files, game backups and other files. It goes without saying that backup reliability is important for server owners.
In November 2018, it became clear to us at Nodecraft that we could improve our costs if we re-examine our cloud backup strategy. After looking at the current offerings, we decided we were moving our backups from Amazon’s S3 to Backblaze’s B2 service. This article describes how our team approached it, why, and what happened, specifically so we could share our experiences.
Due to S3 and B2 being at least nearly equally* accessible, reliable, available, as well as many other providers, our primary reason for moving our backups now became pricing. As we started into the effort, other factors such as variety of API, quality of API, real-life workability, and customer service started to surface.
After looking at a wide variety of considerations, we decided on Backblaze’s B2 service. A big part of the costs of this operation is their bandwidth, which is amazingly affordable.
The price gap between the two object storage systems come from the Bandwidth Alliance between Backblaze and Cloudflare, a group of providers that have agreed to not charge (or heavily discount) for data leaving inside the alliance of networks (“egress” charges). We at Nodecraft use Cloudflare extensively and so this left only the egress charges from Amazon to Cloudflare to worry about.
In normal operations, our customers both constantly make backups as well as access them for various purposes and there has been no change to their abilities to perform these operations compared to the previous provider.
As with any change in providers, the change-over must be thought out with great attention to detail. When there were no quality issues previously and circumstances are such that a wide field of new providers can be considered, the final selection must be carefully evaluated. Our list of concerns included these:
Safety: we needed to move our files and ensure they remain intact, in a redundant way
Availability: the service must both be reliable but also widely available ** (which means we needed to “point” at the right file after its move, during the entire process of moving all the files: different companies have different strategies, one bucket, many buckets, regions, zones, etc)
API: we are experienced, so we are not crazy about proprietary file transfer tools
Speed: we needed to move the files in bulk and not brake on rate limitations, and…
…improper tuning could turn the operation into our own DDoS.
All these factors individually are good and important, but when crafted together, can be a significant service disruption. If things can move easily, quickly, and, reliably, improper tuning could turn the operation into our own DDoS. We took thorough steps to make sure this wouldn’t happen, so an additional requirement was added:
Tuning: Don’t down your own services, or harm your neighbors
What this means to the lay person is “We have a lot of devices in our network, we can do this in parallel. If we do it at full-speed, we can make our multiple service providers not like us too much… maybe we should make this go at less than full speed.”
To embrace our own cloud processing capabilities, we knew we would have to take a two tier approach in both the Tactical (move a file) and Strategic (tell many nodes to move all the files) levels.
Our goals here are simple: we want to move all the files, move them correctly, and only once, but also make sure operations can continue while the move happens. This is key because if we had used one computer to move the files, it would take months.
The first step to making this work in parallel was to build a small web service to allow us to queue a single target file to be moved at a time to each worker node. This service provided a locking mechanism so that the same file wouldn’t be moved twice, both concurrently or eventually. The timer for the lock to expire (with error message) was set to a couple hours. This service was intended to be accessed via simple tools such as curl.
We deployed each worker node as a Docker container, spread across our Docker Swarm. Using the parameters in a docker stack file, we were able to define how many workers per node joined the task. This also ensured more expensive bandwidth regions like Asia Pacific didn’t join the worker pool.
Nodecraft has multiple fleets of servers spanning multiple datacenters, and our plan was to use spare capacity on most of them to move the backup files. We have experienced a consistent pattern of access of our servers by our users in the various data centers across the world, and we knew there would be availability for our file moving purposes.
Our goals in this part of the operation are also simple, but have more steps:
Get the name/ID/URL of a file to move which…
locks the file, and…
starts the fail timer
Get the file info, including size
DOWNLOAD: Copy the file to the local node (without limiting the node’s network availability)
Verify the file (size, ZIP integrity, hash)
UPLOAD: Copy the file to the new service (again without impacting the node)
Report “done” with new ID/URL location information to the Strategic level, which…
…releases the lock in the web service, cancels the timer, and marks the file DONE
The Kill Switch
In the case of a potential run-away, where even the in-band Docker Swarm commands themselves, we decided to make sure we had a kill switch handy. In our case, it was our intrepid little web service–we made sure we could pause it. Looking back, it would be better if it used a consumable resource, such as a counter, or a value in a database cell. If we didn’t refresh the counter, then it would stop all its own. More on “runaways” later.
Real Life Tuning
Our business has daily, weekly, and other cycles of activity that are predictable. Most important is our daily cycle, that trails after the Sun. We decided to use our nodes that were in low-activity areas to carry the work, and after testing, we found that if we tune correctly this doesn’t affect the relatively light loads of the servers in that low-activity region. This was backed up by verifying no change in customer service load using our metrics and those of our CRM tools. Back to tuning.
Initially we tuned the DOWN file transfer speed equivalent to 3/4ths of what wget(1) could do. We thought “oh, the network traffic to the node will fit in-between this so it’s ok”. This is mostly true, but only mostly. This is a problem in two ways. The cause of the problems is that isolated node tests are just that—isolated. When a large number of nodes in a datacenter are doing the actual production file transfers, there is a proportional impact that builds as the traffic is concentrated towards the egress point(s).
Problem 1: you are being a bad neighbor on the way to the egress points. Ok, you say “well we pay for network access, let’s use it” but of course there’s only so much to go around, but also obviously “all the ports of the switch have more bandwidth than the uplink ports” so of course there will be limits to be hit.
Problem 2: you are being your own bad neighbor to yourself. Again, if you end-up with your machines being network-near to each other in a network-coordinates kind of way, your attempts to “use all that bandwidth we paid for” will be throttled by the closest choke point, impacting only or nearly only yourself. If you’re going to use most of the bandwidth you CAN use, you might as well be mindful of it and choose where you will put the chokepoint, that the entire operation will create. If one is not cognizant of this concern, one can take down entire racks of your own equipment by choking the top-of-rack switch, or, other networking.
By reducing our 3/4ths-of-wget(1) tuning to 50% of what wget could do for a single file transfer, we saw our nodes still functioning properly. Your mileage will absolutely vary, and there’s hidden concerns in the details of how your nodes might or might not be near each other, and their impact on hardware in between them and the Internet.
Perhaps this is an annoying detail: Based on previous experience in life, I put in some delays. We scripted these tools up in Python, with a Bourne shell wrapper to detect fails (there were) and also because for our upload step, we ended up going against our DNA and used the Backblaze upload utility. By the way, it is multi-threaded and really fast. But in the wrapping shell script, as a matter of course, in the main loop, that was first talking to our API, I put in a sleep 2 statement. This creates a small pause “at the top” between files.
This ended up being key, as we’ll see in a moment.
How It (The Service, Almost) All Went Down
What’s past is sometimes not prologue. Independent testing in a single node, or even a few nodes, was not totally instructive to what really was going to happen as we throttled up the test. Now when I say “test” I really mean, “operation”.
Our initial testing was concluded “Tactically” as above, for which we used test files, and were very careful in the verification thereof. In general, we were sure that we could manage copying a file down (Python loop) and verifying (unzip -T) and operate the Backblaze b2 utility without getting into too much trouble…but it’s the Strategic level that taught us a few things.
Remembering to a foggy past where “6% collisions on a 10-BASE-T network and its game over”…yeah that 6%. We throttled up the number of replicas in the Docker Swarm, and didn’t have any problems. Good. “Alright.” Then we moved the throttle so to speak, to the last detent.
We had nearly achieved self-DDoS.
It wasn’t all that bad, but, we were suddenly very, very happy with our 50%-of-wget(1) tuning, and our 2 second delays between transfers, and most of all, our kill switch.
TL;DR — Things went great.
There were a couple files that just didn’t want to transfer (weren’t really there on S3, hmm). There were some DDoS alarms that tripped momentarily. There was a LOT of traffic…and, then, the bandwidth bill.
Your mileage may vary, but there’s some things to think about with regards to your bandwidth bill. When I say “bill” it’s actually a few bills.
As per the diagram above, moving the file can trigger multiple bandwidth charges, especially as our customers began to download the files from B2 for instance deployment, etc. In our case, we now only had the S3 egress bill to worry about. Here’s why that works out:
We have group (node) discount bandwidth agreements with our providers
B2 is a member of the Bandwidth Alliance…
…and so is Cloudflare
We were accessing our S3 content through our (not free!) Cloudflare account public URLs, not by the (private) S3 URLs.
Without saying anything about our confidential arrangements with our service partners, the following are both generally true: you can talk to providers and sometimes work out reductions. Also, they especially like it when you call them (in advance) and discuss your plans to run their gear hard. For example, on another data move, one of the providers gave us a way to “mark” our traffic a certain way, and it would go through a quiet-but-not-often-traveled part of their network; win win!
Where Does the Media Industry Really Use Cloud Storage?
Our new cloud survey results might surprise you.
Predicting which promising new technologies will be adopted quickly, which ones will take longer, and which ones will fade away is not always easy. When the iPhone was introduced in 2007, only 6% of the US population had smartphones. In less than 10 years, over 80% of Americans owned smartphones. In contrast, video telephone calls demonstrated at the 1964 New York World’s Fair only became commonplace 45 years later with the advent of FaceTime. And those flying cars people have dreamed of since the 1950s? Don’t hold your breath.
What about cloud storage? Who is adopting it today and for what purposes?
“While M&E professionals are not abandoning existing storage alternatives, they increasingly see the public cloud in storage applications as simply another professional tool to achieve their production, distribution, and archiving goals. For the future, that trend looks to continue as the public cloud takes on an even greater share of their overall storage requirements.”
— Phil Kurz, contributing editor, TV Technology
At Backblaze, we have a front-line view of how customers use cloud for storage. And based on the media-oriented customers we’ve directly worked with to integrate cloud storage, we know they’re using cloud storage throughout the workflow: backing up files during content creation (UCSC Silicon Valley), managing production storage more efficiently (WunderVu), archiving of historical content libraries (Austin City Limits), hosting media files for download (American Public Television), and even editing cloud-based video (Everwell).
We wanted to understand more about how the broader industry uses cloud storage and their beliefs and concerns about it, so we could better serve the needs of our current customers and anticipate what their needs will be in the future.
We decided to sponsor an in-depth survey with TV Technology, a media company that for over 30 years has been an authority for news, analysis and trend reports serving the media and entertainment industries. While TV Technology had conducted a similar survey in 2015, we thought it’d be interesting to see how the industry outlook has evolved. Based on our 2019 results, it certainly has. As a quick example, security was a concern for 71% of respondents in 2015. This year, only 38% selected security as an issue at all.
Survey Methodology — 246 Respondents and 15 Detailed Questions
For the survey, TV Technology queried 246 respondents, primarily from production and post-production studios and broadcasters, but also other market segments including corporate video, government, and education. See chart below for the breakdown. Respondents were asked 15 questions about their cloud storage usage today and in the future, and for what purpose. The survey queried what motivated their move to the cloud, their expectations for access times and cost, and any obstacles that are preventing further cloud adoption.
Survey Insights — Half Use Public Cloud Today — Cloud the Top Choice for Archive
Overall, the survey reveals growing cloud adoption for media organizations who want to improve production efficiency and to reduce costs. Key findings from the report include:
On the whole, about half of the respondents from all organization types are using public cloud services. Sixty-four percent of production/post studio respondents say they currently use the cloud. Broadcasters report lower adoption, with only 26 percent using the public cloud.
Achieving greater efficiency in production was cited by all respondents as the top reason for adopting the cloud. However, while this is also important to broadcasters, their top motivator for cloud use is cost containment or internal savings programs.
Cloud storage is clearly the top choice for archiving media assets, with 70 percent choosing the public cloud for active, deep, or very deep archive needs.
Concerns over the security of assets stored in a public cloud remain, however they have been assuaged greatly compared to the 2015 report, so much so that they are no longer the top obstacle to cloud adoption. For 40%, pricing has replaced security as the top concern.
There’s a wide range of reasons why businesses want to migrate away from their current archive solution, ranging from managing risk, concerns over legacy hardware, media degradation and format support. Many businesses also find themselves stuck with closed format solutions that are based on legacy middleware with escalating support costs. It is a common problem that we at Ortana have helped many clients overcome through smart and effective use of the many storage solutions available on the market today. As founder and CEO of Ortana, I want to share some of our collective experience around this topic and how we have found success for our clients.
First, we often forget how quickly the storage landscape changes. Let’s take a typical case.
It’s Christmas 2008 and a CTO has just finalised the order on their new enterprise-grade hierarchical storage management (HSM) system with an LTO-4 tape robot. Beyonce’s Single Ladies is playing on the radio, GPS on phones has just started to be rolled out, and there is this new means of deploying mobile apps called the Apple App Store! The system purchased is from a well established, reputable company and provides peace of mind and scalability — what more can you ask for? The CTO goes home for the festive season — job well done — and hopes Santa brings him one of the new Android phones that have just launched.
Ten years on, the world is very different and Moore’s law tells us that the pace of technological change is only set to increase. That growing archive has remained on the same hardware, controlled by the same HSM and has gone through one or two expensive LTO format changes. “These migrations had to happen,” the CTO concedes, as support for the older LTO formats was being dropped by the hardware supplier. Their whole content library had to be restored and archived back to the new tapes. New LTO formats also required new versions of the HSM, and whilst these often included new features — over codec support, intelligent repacking and reporting — the fundamentals of the system remained: closed format, restricted accessibility, and expensive. Worse still, the annual support costs are increasing whilst the new feature development has ground to a halt. Sure the archive still works, but for how much longer?
Decisions, Decisions, So Many Migration Decisions
As businesses make the painful decision to migrate their legacy archive, the choices of what, where, and how become overwhelming. The storage landscape today is a completely different picture from when closed format solutions went live. This change alone offers significant opportunities to businesses. By combining the right storage solutions with seamless architecture and with lights out orchestration driving the entire process, businesses can flourish by allowing their storage to react to the needs of the business, not constrain them. Ortana has purposefully ensured Cubix (our asset management, automation, and orchestration platform) is as storage agnostic as possible by integrating a range of on-premises and cloud-based solutions, and built an orchestration engine that is fully abstracted from this integration layer. The end result is that workflow changes can be done in seconds without affecting the storage.
As our example CTO would say (shaking their head no doubt whilst saying it), a company’s main priority is to not-be-here-again, and the key is to store media in an open format, not bound to any one vendor, but also accessible to the business needs both today and tomorrow. The cost of online cloud storage such as Backblaze has now made storing content in the cloud more cost effective than LTO and this cost is only set to reduce further. This, combined with the ample internet bandwidth that has become ubiquitous, makes cloud storage an obvious primary storage target. Entirely agnostic to the format and codec of content you are storing, aligned with MPAA best practices and easily integrated to any on-premise or cloud-based workflows, cloud storage removes many of the issues faced by closed-format HSMs deployed in so many facilities today. It also begins to change the dialogue over main vs DR storage, since it’s no longer based at a facility within the business.
Cloud Storage Opens Up New Capabilities
Sometimes people worry that cloud storage will be too slow. Where this is true, it is almost always due to poor cloud implementation. B2 is online, meaning that the time-to-first-byte is almost zero, whereas other cloud solutions such as Amazon Glacier are cold storage, meaning that the time-to-first-byte ranges from at best one to two hours, but in general six to twelve hours. Anything that is to replace an LTO solution needs to match or beat the capacity and speed of the incumbent solution, and good workflow design can ensure that restores are done as promptly as possible and direct to where the media is needed.
But what about those nasty egress costs? People can get caught off guard when this is not budgeted for correctly, or when their workflow does not make good use of simple solutions such as proxies. Regardless of whether your archive is located on LTO or in the cloud, proxies are critical to keeping accessibility up and costs and restore times down. By default, when we deploy Cubix for clients we always generate a frame accurate proxy for video content, often devalued through the use of burnt-in timecode (BITC), logos, and overlays. Generated using open source transcoders, they are incredibly cost effective to generate and are often only a fraction of the size of the source files. These proxies, which can also be stored and served directly from B2 storage, are then used throughout all our portals to allow users to search, find, and view content. This avoids the time and cost required to restore the high resolution master files. Only when the exact content required is found is a restore submitted for the full-resolution masters.
Multiple Copies Stored at Multiple Locations by Multiple Providers
Moving content to the cloud doesn’t remove the risk of working with a single provider, however. No matter how good or big they are, it’s always a wise idea to ensure an active disaster recovery solution is present within your workflows. This last resort copy does not need all the capabilities of the primary storage, and can even be more punitive when it comes to restore costs and times. But it should be possible to enable in moments, and be part of the orchestration engine rather than being a manual process.
The need to de-risk that single provider, or for workflows where 30-40% of the original content has to be regularly restored (as proxies do not meet the needs of the workflow), on premise archive solutions still can be deployed without being caught in the issues discussed earlier. Firstly, LTO now offers portability benefits through LTFS, an easy to use open format, which critically has its specification and implementation within the public domain. This ensures it is easily supported by many vendors and guarantees support longevity for on-premises storage. Ortana with its Cubix platform supports many HSMs that can write content in native LTFS format that can be read by any standalone drive from any vendor supporting LTFS.
Also, with 12 TB hard drives now standard in the marketplace, nearline based storage has also become a strong contender for content when combined with intelligent storage tiering to the cloud or LTO. Cubix can fully automate this process, especially when complemented by such vendors as GB Labs’ wide range of hardware solutions. This mix of cloud, nearline and LTO — being driven by an intelligent MAM and orchestration platform like Cubix to manage content in the most efficient means possible on a per workflow basis — blurs the lines between primary storage, DR, and last resort copies.
Streamlining the Migration Process
Once you have your storage mix agreed upon and in place, now your fraught task is getting your existing library onto the new solution whilst not impacting access to the business. Some HSM vendors suggest swapping your LTO tapes by physically removing them from one library and inserting them into another. Ortana knows that libraries are often the linchpin of the organisation and any downtime has significant negative impact that can fill media managers with dread, especially since these one shot, one direction migrations can easily go wrong. Moreover, when following this route, simply moving tapes does not persist any editorial metadata or resolve many of the objectives around making content more available. Cubix not only manages the media and the entire transformation process, but also retains the editorial metadata from the existing archive also.
Given the high speeds that LTO delivers, combined with the scalability of Cubix, the largest libraries can be migrated in short timescales, whilst having zero downtime on the archive. Whilst the content is being migrated to the defined mix of storage targets, Cubix can perform several tasks on the content to further augment the metadata, including basics such as proxy and waveform generation, through to AI based image detection and speech to text. Such processes only further reduce the time spent by staff looking for content, and further refine the search capability to ensure only that content required is restored — translating directly to reduced restore times and egress costs.
A Real-World Customer Example
Many of the above concerns and considerations led a large broadcaster to Ortana for a large-scale migration project. The broadcaster produces in-house news and post production with multi-channel linear playout and video-on-demand (VoD). Their existing archive was 3 PB of media across two generations of LTO tape managed by Oracle DIVArchive & DIVADirector. They were concerned about on-going support for DIVA and wanted to fully migrate all tape and disk-based content to a new HSM in an expedited manner, making full use of the dedicated drive resources available.
Their primary goal was to fully migrate all editorial metadata into Cubix, including all ancillary files (subtitles, scripts, etc.), and index all media using AI-powered content discovery to reduce searching times for news, promos /and sports departments at the same time. They also wanted to replace the legacy Windows Media Video (WMV) proxy with new full HD H264 frame accurate proxy, and provide the business secure, group-based access to the content. Finally, they wanted all the benefits of cloud storage, whilst keeping costs to a minimum.
With Ortana’s Cubix Core, the broadcaster was able to safely migrate their DIVAarchive to two storage platforms: LTFS with a Quantum HSM system and Backblaze B2 cloud storage. Their content was indexed via AI powered image recognition (Google Vision) and speech to text (Speechmatics) during the migration process, and the Cubix UI replaced existing archive as media portal for both internal and external stakeholders.
The new solution has vastly reduced the timescales for content processing across all departments, and has led to a direct reduction in staff costs. Researchers report a 50-70% reduction in time spent searching for content, and the archive shows a 40% reduction in restore requests. By having the content located in two distinct geographical locations they’ve entirely removed their business risk of having their archive with a single vendor and in a single location. Most importantly, their archived content is more active than ever and they can be sure it will stay alive for the future.
How exactly did Ortana help them do it?Join our webinarEvading Extinction: Migrating Legacy Archives on Thursday, March 28, 2019. We’ll detail all the steps we took in the process and include a live demo of Cubix. We’ll show you how straightforward and painless the archive migration can be with the right strategy, the right tools, and the right storage.
— James Gibson, Founder & CEO, Ortana Media Group
• • •
Backblaze will be exhibiting at NAB 2019 in Las Vegas on April 8-11, 2019.Schedule a meeting with our cloud storage experts to learn how B2 Cloud Storage can streamline your workflow today!
Whatever your creative venture, the byproduct of all your creative effort is assets. Whether you produce music, images, or video, as you produce more and more of these valuable assets, they tend to pile up and become difficult to manage, organize, and protect. As your creative practice evolves to meet new demands, and the scale of your business grows, you’ll often find that your current way of organizing and retrieving assets can’t keep up with the pace of your production.
For example, if you’ve been managing files by placing them in carefully named folders, getting those assets into a media asset management system will make them far easier to navigate and much easier to pull out exactly the media you need for a new project. Your team will be more efficient and you can deliver your finished content faster.
As we’ve covered before, putting your assets in a type of storage like B2 Cloud Storage ensures that they will be protected in a highly durable and highly available way that lets your entire team be productive.
You can learn about some of the new capabilities of the latest cloud-based collaboration tools here:
With some smart planning, and a little bit of knowledge, you can be prepared to get the most of your assets as you move them into an asset management system, or when migrating from an older or less capable system into a new one.
Assets and Metadata
Before we can build some playbooks to get the most from your creative assets, let’s review a few key concepts.
Asset — a rich media file with intrinsic metadata.
An asset is simply a file that is the result of your creative operation, and most often a rich media file like an image or a video. Typically, these files are captured or created in a raw state, then your creative team adds value to that raw asset by editing it together with other assets to create a finished story that in turn, becomes another asset to manage.
Metadata — Information about a file, either embedded within the file itself or associated with the file by another system, typically a media asset management (MAM) application.
The file carries information about itself that can be understood by your laptop or workstation’s operating system. Some of these seem obvious, like the name of the file, how much storage space it occupies, when it was first created, and when it was last modified. These would all be helpful ways to try to find one particular file you are looking for among thousands just using the tools available in your OS’s file manager.
There’s usually another level of metadata embedded in media files that is not so obvious but potentially enormously useful: metadata embedded in the file when it’s created by a camera, film scanner, or output by a program.
For example, this image taken in Backblaze’s data center a few years ago carries all kinds of interesting information. For example, when I inspect the file on macOS’s Finder with Get Info, a wealth of information is revealed. I can now not only tell the image’s dimensions and when the image was taken, but also exactly what kind of camera took this picture and the lens settings that were used, as well.
As you can see, this metadata could be very useful if you want to find all images taken on that day, or even images taken with that same camera, focal length, F-stop, or exposure.
When a File and Folder System Can’t Keep Up
Inspecting files one at a time is useful, but a very slow way to determine if a file is the one you need for a new project. Yet many creative environments that don’t have a formal asset management system get by with an ad hoc system of file and folder structures, often kept on the same storage used for production or even on an external hard drive.
Teams quickly outgrow that system when they find that their work spills over to multiple hard drives, or takes up too much space on their production storage. Worst of all, assets kept on a single hard drive are vulnerable to disk damage, or to being accidentally copied or overwritten.
Why Your Assets Need to be Managed
To meet this challenge, creative teams have often turned to a class of application called a Media Asset Manager (MAM). A MAM automatically extracts all their assets’ inherent metadata, helps move files to protected storage, and makes them instantly available to their entire team. In a way, these media asset managers become a private media search engine where any file attribute can be a search query to instantly uncover the file they need in even the largest media asset libraries.
Beyond that, asset management systems are rapidly becoming highly effective collaboration and workflow tools. For example, tagging a series of files as Field Interviews — April 2019, or flagging an edited piece of content as HOLD — do not show customer can be very useful indeed.
The Inner Workings of a Media Asset Manager
When you add files into an asset management system, the application inspects each file, extracting every available bit of information about the file, noting the file’s location on storage, and often creating a smaller stand-in or proxy version of the file that is easier to present to users.
To keep track of this information, asset manager applications employ a database and keep information about your files in it. This way, when you’re searching for a particular set of files among your entire asset library, you can simply make a query of your asset manager’s database in an instant rather than rifling through your entire asset library storage system. The application takes the results of that database query and retrieves the files you need.
The Asset Migration Playbook
Whether you need to move from a file and folder based system to a new asset manager, or have been using an older system and want to move to a new one without losing all of the metadata that you have painstakingly developed, a sound playbook for migrating your assets can help guide you.
Play 1 — Getting Assets in Files and Folders Protected Without an Asset Management System
In this scenario, your assets are in a set of files and folders, and you aren’t ready to implement your asset management system yet.
The first consideration is for the safety of the assets. Files on a single hard drive are vulnerable, so if you are not ready to choose an asset manager your first priority should be to get those files into a secure cloud storage service like Backblaze B2.
Then, when you have chosen an asset management system, you can simply point the system at your cloud-based asset storage to extract the metadata of the files and populate the asset information in your asset manager.
Get assets archived or moved to cloud storage
Choose your asset management system
Ingest assets directly from your cloud storage
Play 2 — Getting Assets in Files and Folders into Your Asset Management System Backed by Cloud Storage
In this scenario, you’ve chosen your asset management system, and need to get your local assets in files and folders ingested and protected in the most efficient way possible.
You’ll ingest all of your files into your asset manager from local storage, then archive them to cloud storage. Once your asset manager has been configured with your cloud storage credentials, it can automatically move a copy of local files to the cloud for you. Later, when you have confirmed that the file has been copied to the cloud, you can safely delete the local copy.
Ingest assets from local storage directly into your asset manager system
From within your asset manager system archive a copy of files to your cloud storage
Once safely archived, the local copy can be deleted
Play 3 — Getting a Lot of Assets on Local Storage into Your Asset Management System Backed by Cloud Storage
If you have a lot of content, more than say, 20 terabytes, you will want to use a rapid ingest service similar to Backblaze’s Fireball system. You copy the files to Fireball, Backblaze puts them directly into your asset management bucket, and the asset manager is then updated with the file’s new location in your Backblaze B2 account.
This can be a manual process, or can be done with scripting to make the process faster.
Play 4 — Moving from One Asset Manager System to a New One Without Losing Metadata
In this scenario you have an existing asset management system and need to move to a new one as efficiently as possible to not only take advantage of your new system’s features and get files protected in cloud storage, but also to do it in a way that does not impact your existing production.
Some asset management systems will allow you to export the database contents in a format that can be imported by a new system. Some older systems may not have that luxury and will require the expertise of a database expert to manually extract the metadata. Either way, you can expect to need to map the fields from the old system to the fields in the new system.
Making a copy of old database is a must. Don’t work on the primary copy, and be sure to conduct tests on small groups of files as you’re migrating from the older system to the new. You need to ensure that the metadata is correct in the new system, with special attention that the actual file location is mapped properly. It’s wise to keep the old system up and running for a while before completely phasing it out.
Export the database from the old system
Import the records into the new system
Ensure that the metadata is correct in the new system and file locations are working properly
Make archive copies of your files to cloud storage
Once the new system has been running through a few production cycles, it’s safe to power down the old system
Play 5 — Moving Quickly from an Asset Manager System on Local Storage to a Cloud-based System
In this variation of Play 4, you can move content to object storage with a rapid ingest service like Backblaze Fireball at the same time that you migrate to a cloud-based system. This step will benefit from scripting to create records in your new system with all of your metadata, then relink with the actual file location in your cloud storage all in one pass.
You should test that your asset management system can recognize a file already in the system without creating a duplicate copy of the file. This is done differently by each asset management system.
Export the database from the old system
Import the records into the new system while creating placeholder records with the metadata only
Archive your local assets to Fireball (up to 70 TB at a time)
Once the files have been uploaded by Backblaze, relink the cloud based location to the asset record
Every production environment is different, but we all need the same thing: to be able to find and organize our content so that we can be more productive and rest easy knowing that our content is protected.
These plays will help you take that step and be ready for any future production challenges and opportunities.
If you’d like more information about media asset manager migration, join us for our webinar on March 15, 2019:
If you make copies of your images or video files for safekeeping, are you backing them up or archiving them? It’s been discussed many times before, but the short answer is that it depends on the function of the copy. For media workflows, a crisp understanding is required in order to implement the right tools. In today’s post, we’ll explore the nuances between backup and archiving in media workflows and provide a real world application from UCSC Silicon Valley.
We explored the broader topic of backing up versus archiving in our What’s the Diff: Backup vs Archive post. It’s a backup if you copy data to keep it available in case of loss, while it’s an archive if you make a copy for regulatory compliance, or to move older, less-used data off to cheaper storage. Simple, right? Not if you’re talking about image, video and other media files.
Backup vs. Archive for Professional Media Productions
Traditional definitions don’t fully capture how backup and archive typically operate in professional media workflows compared to business operations. Video and images aren’t typical business data in a number of ways, and that profoundly impacts how they’re protected and preserved throughout their lifecycle. With media backup there are key differences in which files get backed up and how they get backed up. With media archive there are key differences in when files get archived and why they’re archived.
Large Media Files Sizes Slow Down Backup
The most obvious nuance is that media files are BIG. While most business documents are under 30 MB in size, a single second of video could be larger than 30 MB at higher resolutions and frame rates. Backing up such large file sizes can take longer than the traditional backup windows of overnight for incremental backups and a weekend for full backup. And you can’t expect deduplication to shorten backup times or reduce backup sizes, either. Video and images don’t dedupe well.
Meanwhile, the editing process generates a flurry of intermediate or temporary files in the active content creation workspace that don’t need to be backed up because they can be easily regenerated from source files.
The best backup solutions for media allow you to specify exactly which directories and file types you want backed up, so that you’re taking time for and paying for only what you need.
Archiving to Save Space on Production Storage
Another difference is that archiving to reduce production storage costs is much more common in professional media workflows than with business documents, which are more likely to be archived for compliance. High-resolution video editing in particular requires expensive, high-performance storage to deliver multiple streams of content to multiple users simultaneously without dropping frames. With the large file sizes that come with high-resolution content, this expensive resource fills up quickly with content not needed for current productions. Archiving completed projects and infrequently-used assets can keep production storage capacities under control.
Media asset managers (MAMs) can simplify the archive and retrieval process. Assets can be archived directly through the MAM’s visual interface, and after archiving, their thumbnail or proxies remain visible to users. Archived content remains fully searchable by its metadata and can also be retrieved directly through the MAM interface. For more information on MAMs, read What’s the Diff: DAM vs MAM.
Strategically archiving select media files to less expensive storage allows facilities to stay within budget, and when done properly, keeps all of your content readily accessible for new projects and repurposing.
Permanently Secure Source Files and Raw Footage on Ingest
A less obvious way that media is different is that video files are fixed content that don’t actually change during the editing process. Instead, editing suites compile changes to be made to the original and apply the changes only when making the final cut and format for delivery. Since these source files are not going to change, and are often irreplaceable, many facilities save a copy to secondary storage as soon as they’re ingested to the workflow. This copy serves as a backup to the file on local storage during the editing process. Later, when the local copy is no longer actively being used, it can be safely deleted knowing it’s secured in the archive. I mean backup. Wait, which is it?
Whether you call it archive or backup, make a copy of source files in a storage location that lives forever and is accessible for repurposing throughout your workflow.
To see how all this works in the real world, here’s how UCSC Silicon Valley designed a new solution that integrates backup, archive, and asset management with B2 cloud storage so that their media is protected, preserved and organized at every step of their workflow.
How UCSC Silicon Valley Secured Their Workflow’s Data
UCSC Silicon Valley built a greenfield video production workflow to support UC Scout, the University of California’s online learning program that gives high school students access to the advanced courses they need to be eligible and competitive for college. Three teams of editors, producers, graphic designers and animation artists — a total of 22 creative professionals — needed to share files and collaborate effectively, and digital asset manager Sara Brylowski was tasked with building and managing their workflow.
Sara and her team had specific requirements. For backup, they needed to protect active files on their media server with an automated backup solution that allowed accidentally deleted files to be easily restored. Then, to manage storage capacity more effectively on their media server, they wanted to archive completed videos and other assets that they didn’t expect to need immediately. To organize content, they needed an asset manager with seamless archive capabilities, including fast self-service archive retrieval.
They wanted the reliability and simplicity of the cloud to store both their backup and archive data. “We had no interest in using LTO tape for backup or archive. Tape would ultimately require more work and the media would degrade. We wanted something more hands off and reliable,” Sara explained. The cloud choice was narrowed to Backblaze B2 or Amazon S3. Both were proven cloud solutions that were fully integrated with the hardware and software tools in their workflow. Backblaze was chosen because its $5 per terabyte per month pricing was a fraction of the cost of Amazon S3.
Removing Workflow Inefficiencies with Smarter Backup and Archive
The team had previously used the university’s standard cloud backup service to protect active files on the media server as they worked on new videos. But because that cloud backup was designed for traditional file servers, it backed up everything, even the iterative files generated by video production tools like Adobe Premiere, After Effects, Maya 3D and Cinema 3D that didn’t need to be backed up. For this reason, Sara pushed to not use the university’s backup provider. It was expensive in large part because it was saving all of this noise in perpetuity.
“With our new workflow we can manage our content within its life cycle and at same time have reliable backup storage for the items we know we’re going to want in the future. That’s allowed us to concentrate on creating videos, not managing storage.”—Sara Brylowski, UCSC Silicon Valley
After creating thousands of videos for 65 online courses, their media server was quickly filling to its 128 TB capacity. They needed to archive data from completed projects to make room for new ones, sooner rather than later. Deploying a MAM solution would simplify archiving, while also helping them organize their diverse and growing library of assets — video shot in studio, B-roll, licensed images, and audio from multiple sources.
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.