Back-ups only help if they are actually working.
May 14, 2012 8:34 AM Subscribe
This post was deleted for the following reason: If it's any comfort, this double post isn't as bad as Cars 2. -- cortex
I feel like I spent that two and a half minutes rather well. Thanks!
posted by Shutter at 8:47 AM on May 14, 2012
posted by Shutter at 8:47 AM on May 14, 2012
When I was up at Pixar for a meeting, the CIO told the story to me. Here's how I heard it.
Someone mistakenly was doing clean up and did an rm -rf in the main render instructions directory. The way Pixar animation works is that the creative tools are used to make the instructions that a very large (3,000+ node at the time) render farm uses to generate the actual video footage. These instructions are not that big in comparison to video footage, kind of like the script of a movie in a file as compared to the final video file.
As soon as the person realized what they had done, they called the IT group immediately. But by that point, there was nothing that could have been done. They went to the back-ups and found that they didn't have complete back-ups due to using an older method of back-up and also not checking the tapes to see if they were still readable. When magnetic tapes are used for back-up, they can be recorded over many, many times, and can end up in a state where either you can't read them back. Most modern back-up infrastructures now require verification of data recovery on separate hardware, but this was not the case at the time.
One of the producers had been working at home and had most all of the film as render instructions stored on her laptop. They were able to rebuild the data set for the most part and lose very little.
Needless to say, after that, the CIO was cleared to build the most foolproof back-up and archival system I had ever seen. Back-ups now occur at every level from the workstation to the various libraries and render farms. If one layer fails, there are other layers that can be accessed. Returning to my office, I advocated implementation of a similar back-up paradigm, but was turned down due to cost.
I ride bikes with the former CIO, and I'll ask him the next time we get out for more details.
posted by Argyle at 8:51 AM on May 14, 2012 [9 favorites]
Someone mistakenly was doing clean up and did an rm -rf in the main render instructions directory. The way Pixar animation works is that the creative tools are used to make the instructions that a very large (3,000+ node at the time) render farm uses to generate the actual video footage. These instructions are not that big in comparison to video footage, kind of like the script of a movie in a file as compared to the final video file.
As soon as the person realized what they had done, they called the IT group immediately. But by that point, there was nothing that could have been done. They went to the back-ups and found that they didn't have complete back-ups due to using an older method of back-up and also not checking the tapes to see if they were still readable. When magnetic tapes are used for back-up, they can be recorded over many, many times, and can end up in a state where either you can't read them back. Most modern back-up infrastructures now require verification of data recovery on separate hardware, but this was not the case at the time.
One of the producers had been working at home and had most all of the film as render instructions stored on her laptop. They were able to rebuild the data set for the most part and lose very little.
Needless to say, after that, the CIO was cleared to build the most foolproof back-up and archival system I had ever seen. Back-ups now occur at every level from the workstation to the various libraries and render farms. If one layer fails, there are other layers that can be accessed. Returning to my office, I advocated implementation of a similar back-up paradigm, but was turned down due to cost.
I ride bikes with the former CIO, and I'll ask him the next time we get out for more details.
posted by Argyle at 8:51 AM on May 14, 2012 [9 favorites]
« Older Shake & Fold | Build the Enterprise Newer »
This thread has been archived and is closed to new comments
posted by zamboni at 8:45 AM on May 14, 2012 [1 favorite]