Delete? Are you sure?

First published on Imps in the Archive on 20 September 2018.

One way to save an old movie is to bury it in a disused swimming pool in the frozen Yukon.

That’s what builders in Dawson City, Alaska, found when they excavated a derelict site in 1978 and unearthed a cache of hundreds of old films. The cold climate had preserved the nitrate film stock which, along with other ‘rubbish’, had been used to fill in the swimming pool when an ice rink was constructed on top of it. Among the 533 reels were scenes from movies for which there are no other existing copies and snippets of long-lost history, like footage of the 1919 World Series. [1]

Following their discovery the films were extracted and preserved, and now the fragments have been woven together into a documentary. [2]

It’s one thing for a cinema in a small town to carelessly dispose of the films it no longer needs. But one modern film company almost deleted a movie – the second in one of the successful franchises of all time – while it was still making it.

The animated film Toy Story, released in 1995, was the first feature length film created in CGI (computer generated imagery) [3] and was an instant smash hit.

People loved the adventures of the toys – Buzz Lightyear the spaceman, Woody the cowboy and their friends – so much that the film was nominated for three Oscars and, since release, has earned more than $370 million worldwide [4].

So before too long, attention turned to Toy Story 2. A release date of November 1999 was set and work began at Pixar Animation Studios. The work proceeded, the scenes came together and it seemed to be on track.

Then one afternoon in 1998 Woody the cowboy started to disappear….

Oren Jacob , then an associate technical director for Toy Story 2, recalled years later: “That’s when we first noticed it, with Woody.” Larry [Cutler, an associate technical director] was in that directory and happened to be talking about installing a fix to Woody or Woody’s hat. He looked at the directory and it had like 40 files, and he looked again and it had four files.” [5]

We all know that sinking feeling of losing an important file even when it’s not a multi-million dollar movie.

“[Larry] had an error, I forget the exact [one]. “It was like, “Directory no longer valid,” because he’s in a place that had just been deleted. Then he thought to walk up [a directory] and he walked back up and then we saw Hamm, Potato Head and Rex. Then we looked at it again and there was just Hamm and then nothing.” [6]
Oren Jacob, 2012

It seemed that someone at the company, intended to delete one folder, had set the ‘remove’ command in motion and it was working its way through the whole server, deleting file after file, and folder after folder.

According to Matthew Panzarino, who interviewed Oren in 2012, “Unfortunately, someone on the system had run the command at the root level of the Toy Story 2 project and the system was recursively tracking down through the file structure and deleting its way out like a worm eating its way out from the core of an apple.” [7]

In a panic, Pixar pulled the plug on the network connection to the server but it wasn’t soon enough. But when they brought the machine back up, 90% of the movie had been deleted. [8]

But at least they had the back-ups, right? Yes!

So they got them out of the back-up file, and after a few deep breaths, got back to work. Only to find that the back-ups weren’t quite as backed-up as they’d thought…

“…we had restored the film from backups within 48 hours of the /bin/rm -r -f * [the remove command], run some validation tests, rendered frames, somehow got good pictures back and no errors, and invited the crew back to start working. It took another several days of the entire crew working on that initial restoral to really understand that the restoral was, in fact, incomplete and corrupt. Ack. At that point, we sent everyone home again and had the come-to-Jesus meeting where we all collectively realized that our backup software wasn’t dishing up errors properly (a full disk situation was masking them, if my memory serves), our validation software also wasn’t dishing up errors properly (that was written very hastily, and without a clean state to start from, was missing several important error conditions), and several other factors were compounding our lack of concrete, verifiable information…. The only prospect then was to roll back about 2 months to the last full backup that we thought might work.”
Oren Jacob on Quora, 2012 [9]

It was at this crunch point that Supervising Technical Director Galyn Susman remembered her home computer. When she’d had a baby her personal machine was hooked up to the Pixar network so she could work at home. This meant it had been getting regular updates of the Toy Story files.

After a nervous journey to retrieve and examine her computer they copied off a two-week old back-up, to combine it with the files in the two-month old back-up at the office and a few other random pieces of work that the animators had kept in personal folders. They retrieved 70,000 current files but that still left 30,000 files to be compared manually – a task that a team of people worked round-the-clock over a weekend to complete.

Their efforts paid off and Toy Story 2 (after a few more creative travails and a remake) was in cinemas on the appointed date and went on to take $497million worldwide [10].

What are the lessons?

That companies should allow employees to keep copies of precious business assets at home, just in case something goes wrong…?

No!

It was very handy that they could get the copies off Susman’s home computer. But it’s not really good practice for employees to keep copies of work files on machines that are probably less secure than the company’s computers.

A better outcome would be to configure the system so that no-one has the capacity to delete every folder. That might mean developing work processes to move items in and out of shared folders which have different editing permissions depending on the stage of development. That may be irksome but those processes could save a lot of stress and money.

In both these scenarios luck played a huge part.

The Dawson City cinema had no business reason to keep their 1920s film reels but official archiving institutions didn’t preserve them either. We now have those films because of accidental preservation.

Toy Story 2 could have been recreated but it would have taken the team months with huge financial implications for the company if it had missed its release date; the film’s survival in that form was due to the incidental preservation in the home computer.

Pixar’s back-up system had run out of space so it was deleting older files to save newer ones. Not only that but there wasn’t even space in the system to send the warning message that the back-up space was full. What the home computer was doing was making an ‘unofficial’ back-up, an activity that which turned out to be more reliable than the corporate process.

So the big lesson is that it’s essential to back up your files. But not only that; you also need to regularly test that the back-up system is working.

However it all ended happily and now it even makes a good story. So good, in fact, that they made it into a little movie!