It’s bound to happen to every videographer at some point in their career: The shoot is wrapped, the last SD card has been pulled from the camera and the Data Wrangler on the set turns ashen and says to you, “Wait a minute, there’s nothing on this card!” Sure enough it’s Miller time but for all the wrong reasons.
The loss of video data can be a game (if not career) changer on many levels. But rather than get drunk, point fingers, blame your camera operator, or have a full-fledged psychotic meltdown, let’s talk about ways you can avoid the data loss, and what you can do about it should you encounter such an unfortunate occurrence!
While recording, most cameras write their video data through to storage but keep metadata and indices in their internal memory. When you stop recording, your camera saves that metadata, along with indices, at the end of the recorded file.
If something unexpectedly interrupts the recording process, your camera may fail to save all of the important metadata and indices with your video file and the card will appear empty.
What could interrupt the recording process you might ask? You may have accidentally dropped your camera; or failed to stop recording before you powered down: These are just two examples but there are many more including, if the battery suddenly dies while you are still recording. Ouch!
Be very careful handling your data as there are any numbers of causes that can result in a video file being either lost or corrupted. So how can you avoid losing your data? That’s where having a skilled DIT on-location becomes critical.
The DIT (digital image technician) is often considered a valued component to the camera crew although their skills can often crossover into post-production as well, especially on lower budget projects.
Often referred to as a “data wrangler,” the DIT’s role can be much more than that, often being tasked with color grading and/or color correction of the video on location. Working closely with the cinematographer in more or less real time, the DIT serves to maintain image quality and data integrity throughout the production process. Needless to say, a DIT can play a very important role whether on-set or on-location.
With experience often comes an awareness of what’s best to avoid and sometimes the most important thing to avoid is complexity.
A few years ago, we at Blare Media were shooting for three days on location for a major client that sells hamburgers and is fond of arches: This particular time the DIT on location chose to set up the hard drives in the RAID in a novel configuration.
What seemed like a good idea at first became a problem by the end of the multi-day shoot when the RAID failed for unknown reasons. Fortunately, no data was lost but it led to a situation where most of the cast and crew ended up sitting around waiting for the drives to finish backing up. Since the producer was not about to leave the location without all the data backed up, there were a tense few hours that resulted in quite a bit of unexpected overtime for cast and crew.
In theory, the data backup concept chosen wasn’t a bad one but it did have more points of failure than a more traditional approach and perhaps more importantly, it appeared to the client’s producer that Blare Media wasn’t using the proven “tried and true” method.
Had our DIT chosen to go with the simpler, more traditional route, then the backlash from the producer might have been, “The drive failed, that happens, and that’s why we have a backup. We can wait:” But working with an “outside the box” approach, our DIT put Blare Media on the hook financially, something we must try to avoid at all times!
Sometimes we make solutions in search of a problem that can get us into a lot of trouble.
So how does Blare Media’s DIT manage a client’s data today?
Once a camera card is full, and while we’re still on location, the DIT offloads the data onto two separate SSDs. What is an SSD you may ask? An SSD (solid state drive) is a large digital storage device similar to a hard drive but since SSDs lack the moving parts of a typical hard drive, they are often viewed as more reliable and less susceptible to shocks than most hard drives. This makes them more resistant to mishandling. Note that we said “resistant” to mishandling, no drive available on the market today has been proven to be fully idiot-proof.
But let’s get back to Blare Media’s process again: Only when the camera cards are fully double-checked and the media is 100% copied and verified does the DIT consider reformatting the camera cards for reuse.
Meanwhile, Blare Media creates a new footage folder with a unique ID on their RAID 6. A RAID 6 server (redundant array of independent disks) is one of multiple RAID configurations used in the storage of video files across several disks, thus providing data on multiple drives for additional file failure protection.
Blare’ DIT then transfers one of the two SSD drives to the RAID 6 server using verification checksum for error detection and verification of data integrity since data errors may have been introduced during any step in the recording, transmission or storage processes.
After the media is copied, checked and double-checked, Blare then tests the files to ensure there is no corruption before beginning the post-production process. If errors or damaged files are found, they may still be recovered: We’ll talk about more on that later in the blog.
When the process is completed, all media for that project is secured on the server prior to the last step, which is moving all of the files onto Blare’s long-term backup system using LTO tapes.
Why use an LTO tape and not a hard drive? The answer to this question comes in two parts: First, while a normal hard drive’s useful life is relatively short, an LTO (linear tape open) system is capable of storing large amounts of data between 15 and 30 years if stored under optimum conditions. This reassures Blare’s clients that their files will be safely stored for many years and available for use in the future. Second, there is no product available anywhere in the world that is cheaper than tape for long-term data storage. So the short answer is; LTO tapes provide longevity and economy for storing your important data.
Only after the LTO backup is completed does Blare Media consider removing the files from our server, the SSD drives and from the remaining camera cards before being repurposed for work on the next project.
So what happens if and when a file is corrupted? We’ve found that often times the data can be recovered and it’s not as difficult as it might seem at first.
At Blare Media we have learned that Restore.Media can often repair corrupted, damaged and unplayable video or audio files recorded on most any camera, drone or mobile device in a number of different formats.
For example, if you’ve deleted a video file by mistake, restored it but still find the file is not playable, you can very likely call on Restore.Media to repair a valid part of that video file.
Another example, if you have an unplayable video file with broken headers, indices and missing important metadata and find it can’t be played back; Restore.Media can repair such video in most cases.
However, there are times when the file is so severely fragmented or damaged and contain junk fragments. If this is the case, you could make a backup image file (.img) of the flash memory card, from which you’ve deleted the video file, upload it to cloud storage, and send the link via e-mail to firstname.lastname@example.org
Restore.Media can repair such video files in most of the cases: However, what they can’t repair are distortions, glitches or freezes because this may be how your camera actually recorded the video data.
Unfortunately, such video files cannot be fixed since this could be the result of a malfunction either in your camera or your SD card. To prevent future problems, make sure you use a proper memory card that is fast enough to handle your camera’s bit-rate settings.
To wind this up, at Blare Media email@example.com we have worked diligently for many years learning (sometimes the hard way) what it takes to avoid losing or damaging our clients valuable video data and how to recover data that may have otherwise been lost.
So take a page from Blare Media’s book and learn from our mistakes. Avoid the pitfalls if you can but if it’s too late for that, we suggest you visit restore.media to try and let them try and resolve your problems. Good luck!