A Confederacy of Redundencies

​by Peter Schneider

Data management is an increasingly important part of a sound mixer’s job  Until they hand the files off, the mixer often has the only copies of the production audio; if there’s something wrong with it, half the project is gone, so back up systems for redundancy is key. But sometimes, even with back up plans, the worst does happen, so we occasionally are called upon to perform data recovery on field audio recordings.

We got one case right before Christmas that got me thinking about the nature of field recording and how we can do it more reliably. This particular customer was recording location audio for a reality television show for five hours into a recorder that records simultaneously to a hard drive disk (HDD) and CompactFlash (CF) card. When they went to transfer the files, they discovered that they weren't anywhere to be found; both mediums essentially looked like they had just been formatted by the machine. They were particularly distressed because they didn't understand how they ended up with no audio at all - they felt that since they were recording to both a HDD and a CF card that they had redundant recordings.

Many years ago, I discovered the book Normal Accidents by Charles Perrow. One of his central themes is that mishaps in complex systems happen in part when components are "tightly coupled” - in other words, when one part depends on the other to operate properly. In the case of a recorder that writes to two separate media, you may be protected from media error or loss, but you are not protected if, for example, a media error causes the machine to lock up. This system is said to be tightly coupled because a failure in one sub-part - a media error that causes a lock up, a power failure, etc., will cause a failure in another sub-part.

When designing recording systems, it's important to look for these single points of failure. In the above example, maybe having only one recorder is an acceptable risk. These recorders are basically purpose-built laptops - every bit as reliable and every bit as complex. Perhaps it’s an acceptable trade-off to forgo the weight and expense of having a second recorder knowing that the "backup" audio is being recorded as a mix track wirelessly to camera. In other situations, it would be unacceptable to walk away without the multi-track recordings and so a second recorder needs to be added. But each link in the audio, power and sync signals must be scrutinized for evidence of tight coupling and the risk of a single point of failure must be weighed against the cost of eliminating it.

When a control surface is used, additional thought must be given to how best to back up the audio, since the mix is now dependent on the recorder coupled to the control surface. Tests must be performed to ensure that audio fed to a second recorder won't be interrupted if the first recorder and/or control surface fails.

​All of this was swimming around my post-holiday brain when I came across a post by my colleague Benoit Jooson from Aero Quartet. (First, a word about Aero Quartet: They specialize in recovering video files from cameras and recorders that have gone haywire. They are VERY good and have incredibly consumer-friendly pricing and service that is most welcome during a stressful time).

Benoit offers fantastic advice for creating truly redundant recording recordings:

1. No “Back Up Recorder”
   Don't think of your back up recorder as a back up; think of it as an alternate prime. If you only pay attention to your back up recording when it's needed, how do you know it's really working?
    
2. Diverse Workflows
   Independent recordings must have separate paths, but also can't be susceptible to the same issues. His example is a bug that exists in identical machines can cause both machines to stop recording. If you want true separate recording, consider using different models of machines, possibly even from different manufacturers.
    
3. Supervision
   A back up is only good if it's monitored. For example, if you’re using a wireless send to camera as a back up for the mix audio and the camera operator isn't listening to the audio, there will be no way to detect RF interference or distorted levels.

Benoit's suggestion of forgetting about the concept of primary and back up recorders and instead thinking about both recorders as being the primary recorder on alternate days is genius. If you are responsible for delivering location audio, I urge you to read his entire article.

Incidentally, we were able to recover our customer's audio using a tool called R-Tools. Interestingly, once the files were unhidden, the hard drive contained gibberish audio, while the CF card contained the actual production audio. Something catastrophic happened early on in the recording process which caused neither media to close properly, but at least the audio existed on the CF card.

Want to talk about designing a system to keep your recording as safe as possible? Feel free to get in touch with us; we’re happy to work with you on a solution that fits your budget, space, and workflow.