Pixar almost deleted Toy Story 2 (2012)

https://thenextweb.com/news/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good

by feross • 4 years ago

181 91 4 years ago

dingosity2• 4 years ago

This is a great example of why it's a good idea to always test your backups.

ltbarcly3• 4 years ago

I came to make this snide comment as well! Nobody tests their backups though, I've only ever seen it done inadvertently when the backups were used as part of the development process.

yakubin• 4 years ago

BTW how do you test a personal workstation backup? I don’t have backup workstations to rotate between by backing one up, restoring on the other, working and then cycling between them in this manner. How do you test the restore procedure without risking destruction of the original?

vikingerik• 4 years ago

You don't need backups. You need restores.

haswell4• 4 years ago

Generally speaking, I think this is what testing a backup means.

pstuart• 4 years ago

In the olden days when I was the admin for a small shop we would immediately restore the backup to a second system to be used as a playground. Win/win: verified restore and a place to test/train with production data.

sixothree• 4 years ago

Yup, just as long as none of the connection strings point to (and can reach) a prod database.

brainwipe• 4 years ago

It's 2022. Test your backups.

haswell4• 4 years ago

While yes, backups should be tested, this all happened over two decades ago.

DefNotMe• 4 years ago

Is kottke.org operated by Daniel Kottke?

jonhohle• 4 years ago

Jason Kottke, unrelated - https://kottke.org/99/06/dan-kottke

jefftk• 4 years ago

More details on Quora: https://www.quora.com/Pixar-company/Did-Pixar-accidentally-d...

The ending is sad -- that intrepid recovery mostly didn't end up used:

And then, some months later, Pixar rewrote the film from almost the ground up, and we made Toy Story 2 again. That rewritten film was the one you saw in theatres and that you can watch now on Blu-Ray.

haswell4• 4 years ago

I don't know about sad necessarily - if the earlier iteration was necessary to realize they needed to make the changes that brought us the final version, it's all part of a larger journey. I can see how this could be frustrating for people involved, but reshoots / rewrites are part of the process.

And I suspect that the recovery of character models and other assets was still instrumental in making the final film, if not the actual shots/sequences themselves.

trynewideas• 4 years ago

Considering how good Toy Story 2 was, "sad" is an odd way of putting it, unless you're talking about the ensuing crunch.

There are a few more details about that side of the story elsewhere:

https://www.independent.co.uk/arts-entertainment/films/news/...

https://www.denofgeek.com/movies/toy-story-2-disasters-almos...

nathanvanfleet2• 4 years ago

The sad part is that the thing they saved didn't actually get used and was just thrown out. So it wasn't exactly the big save it's made up to be. But if it wasn't framed as a big save maybe it wouldn't be posted every week or so.

diskzero8• 4 years ago

I think we made How to Train Your Dragon three times at DreamWorks. So many rewrites, director changes, character redesign, etc. If you ever wonder why a feature length animated film can cost over $300 million dollars, this is one reason.

pkulak• 4 years ago

Worth it. Love that movie.

mmcgaha• 4 years ago

It is good to hear that big up front design sucks as much for movies as it does for software.

skunkworker• 4 years ago

In college I remember hearing from one of my animation professors that DreamWorks switched between NURBS and polygons for How To Train Your Dragon, I'm unsure which direction it was but I don't envy the artists tasked with that if true.

davidferguson2• 4 years ago

Oh you worked on How to Train Your Dragon? That's my favourite movie in the world!

Really random follow up question - there's a How to Train Your Dragon advert that I assume played before the first film in cinemas, advertising the cinema's popcorn and other concessions, but the only two copies on YouTube are in really terrible quality. (one in English https://www.youtube.com/watch?v=f37fI14Ne0I, the other in Finnish I think https://www.youtube.com/watch?v=W0Hr4mD3G1Q). I really really want a high quality version of this, and I'm wondering if you know of anyone at DreamWorks who I can contact about this? I've tried tweeting at a couple of people but unsurprisingly heard nothing back, so I'm maybe looking for someone in marketing?

coding123• 4 years ago

The dragon looks exactly like one of my cats. Thank you for iterating.

hinkley5• 4 years ago

Thanks for nothing, you useless reptile.

Seriously though, did they have to record the voice acting three times, or partially again several times?

Daub2• 4 years ago

Nice to hear this story.

I teach in an animation program. We tell our students over and over that animation is not like the movies: re-=shoots, pick-up shots, multiple take etc are simply not possible. That is why the bloopers reel in Toy Story was so ironic.

AdmiralAsshat• 4 years ago

This seems to be par for the course for Pixar. I remember many of the deleted scenes from the Inside Out DVD were prefaced with explanation from Pixar that they were from a much earlier draft. They realized after working through the first draft that they now had a much better understanding of the characters, and then rewrote the narrative to match.

chx• 4 years ago

Zootopia was also rewritten. https://lostmediawiki.com/Zootopia_(partially_found_pre-rewr...

This decision came fairly late in the process, as animators had been working on the film for almost a year with only 17 months until it needed to be completed.

geocrasher• 4 years ago

My favorite part of this account isn't the "test your backups!" lesson. It's the "Have an amazing team" lesson. In one weekend, a team of actual humans sorted through thirty thousand files by hand.

Surround yourself with amazing people, and you will find amazing solutions.

outside1234• 4 years ago

and then make sure those amazing people are testing their backups. :)

bluedino• 4 years ago

> Galyn mentioned she might have a copy at her house. So we went home to get that machine, and you can watch the video for how that went…

DLP software hadn't yet been invented

helsinkiandrew• 4 years ago

I think there’s more background and detail in this version:

https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...

dang5• 4 years ago

We've changed the URL to that from https://kottke.org/12/05/how-pixar-almost-deleted-toy-story-.... Thanks!

naikrovek2• 4 years ago

this is insane to me because individual frames for a Pixar film are around 1GB in size.

recall how large laptop hard drives were before Toy Story 2 (2002)...

I have an extremely small amount of experience writing shaders for RenderMan (BMRT, actually), and those aren't large. animation data typically isn't HUGE, but models sure as heck are, and a frame you send to RenderMan is concrete (it used to be, maybe it isn't anymore?) meaning that all geometry was present in the file you gave to RenderMan, only the compiled shaders were referenced from it, because they were procedural and could not be statically included. so I guess the frames the renderer sees are sort of like the binaries of what I would now view as a software compilation process.

raldi2• 4 years ago

I think this story is about the models, not the rendered frames.

naikrovek2• 4 years ago

well that's my point. I always thought that what was lost were frame definition files, and not the models. surely every animator had copies of the models; you can't animate without them, and the movie was in production when the data was lost, so they had those models, or certainly most of them.

I think what was lost was the animations, and not the models or the frame/scene files.

raldi2• 4 years ago

I'm reminded of the time I was on the InfoSec team at a company you've all heard of where all software development was done on a handful of Unix servers with a shared network file system.

I reached out to the Systems team to suggest they look into mounting the home directories with the no-setuid option, and one of them invited me over to pair at their desk. They explained that there was a trick, where you could leave the existing mount point as is, with its existing mount options, and then mount the same device somewhere else with different options. So they got a root shell, went into some scratch directory, mounted the homedirs there, and we tested things out.

I forget how that went, or what happened next, but it seems that we neglected to clean up after this experiment, and at some point, another Systems engineer was looking around the scratch directory and perhaps ran something like `du -ak` (forgetting to specify the option that disables crossing onto new devices), finding what appeared to be a gigantic copy of everyone's home directories.

So they started `rm -rf`'ing it, and made a pretty huge dent before the phones started ringing and they realized that what they were erasing was not a copy on a local scratch space, but rather the real thing over on the NFS drive.

Fortunately, this company did have great backups.

LanceH• 4 years ago

I was one ROLLBACK away from the Users table in production. Mid 90's, we fixed things in production all the time, and I left the where clause out. There were no soft deletes back then, we just deleted the row, and that would trigger the deletion of user resources elsewhere...

I'm sure we had backups, but it would have been bad and resulted in a lot more fixes in production. I guess the upside of the "seat of the pants" days is that there was poor referential integrity, so it wouldn't have taken out anything else like billing history.

edit: I never typed anything so carefully as I typed that rollback. I almost want to say there was an autocommit when closing the session, but my memory on that is fuzzy. MS SQL Server, if anyone knows. From then on, when called on for sketchy but necessary actions, I write the where clause first, run it as a select, see the number of results, wrap it in a transaction, etc...

tomcam• 4 years ago

Good story although there were indeed soft deletes in the 90s.

knodi123• 4 years ago

I was manually monitoring a slow query of mine with "SHOW FULL PROCESSLIST", and at one point I saw "delete from readings" pop up. Readings being a massive, massive table that drove most of our functionality. I quickly killed the query and went about trying to see who else was connected to the DB, but then I saw the query come up again. So I killed it again, and second later it came back.

I had to shout one of the sales guys over, and tell him to go office to office and demand anybody who was deleting rows to stop. Meanwhile I stayed at my desk and kept killing the query.

Shortly after, I heard our CTO yell "shit!", and the query went away. He was trying to clear out his local testing DB, and didn't realize he was in a production shell. I don't know if I saved the company, but I certainly saved us from a couple of days of downtime and maybe an equal or larger amount of lost data. But the CTO couldn't praise me publicly or he'd look like an idiot, so that was the end of that story!

dang5• 4 years ago

A baby saved ‘Toy Story 2’ from near complete deletion (2021) - https://news.ycombinator.com/item?id=30330740 - Feb 2022 (121 comments)

Toy Story 2 files wiped by accident, restored thanks to work-from-home employee - https://news.ycombinator.com/item?id=23022717 - April 2020 (2 comments)

How Pixar’s Toy Story 2 was deleted twice (2012) - https://news.ycombinator.com/item?id=20271179 - June 2019 (2 comments)

Toy Story 2 Got Deleted Twice, Once on Accident, Again on Purpose (2012) - https://news.ycombinator.com/item?id=20265328 - June 2019 (4 comments)

How One Line of Text Nearly Killed Toy Story 2 - https://news.ycombinator.com/item?id=16621848 - March 2018 (1 comment)

How Pixar’s Toy Story 2 was deleted twice [] (2012) - https://news.ycombinator.com/item?id=15978145 - Dec 2017 (1 comment)

Did Pixar accidentally delete Toy Story 2 during production? (2012) - https://news.ycombinator.com/item?id=13417037 - Jan 2017 (253 comments)

How Toy Story 2 Got Deleted Twice, Once on Accident, Again on Purpose - https://news.ycombinator.com/item?id=11500467 - April 2016 (1 comment)

Rm -rf nearly killed Toy Story 2 - https://news.ycombinator.com/item?id=11326831 - March 2016 (1 comment)

How Toy Story 2 was nearly lost - https://news.ycombinator.com/item?id=9739675 - June 2015 (1 comment)

Pixar’s Toy Story 2 was deleted twice (2012) - https://news.ycombinator.com/item?id=8786790 - Dec 2014 (2 comments)

How Pixar’s Toy Story 2 was deleted twice once by tech & again for its own good - https://news.ycombinator.com/item?id=4001714 - May 2012 (1 comment)

How Pixar Almost Lost Toy Story 2 To A Bad Backup - https://news.ycombinator.com/item?id=3972798 - May 2012 (75 comments)

onionisafruit• 4 years ago

It looks like you're saying you've had this conversation before.

dang5• 4 years ago

'Related' lists are just for people who might be curious to read past related threads.

(Arguably this one is dupey because the last major discussion was already this year, but I decided to let it slide in this case)

joezydeco3• 4 years ago

I used to flag duplicates from the new page, but realized it goes nowhere. Maybe a different flag to downgrade dupes?

ChrisArchitect• 4 years ago

Yes please.

This is hardly news, even if it prompts discussion, most of it is repeating the same discussion from all the previous discussions

nathanvanfleet2• 4 years ago

I kind of wonder if deduping posts would be good. I find it really annoying to see reposts and if definitely makes it feel like there really isn't all that much content out there, at least much content that has enough viral potential. I suppose it's possible for a company to fingerprint content easily enough to stop dupes. Maybe it would block the post but allow a repost of the original content? But is that really good for the company? Plenty of people haven't seen that content anyway, so maybe it's working for those people, and sometimes you do want to see the repost if it's been long enough. I personally think it might just seize the entire ecosystem if you prevented it because the volume of content would go down, and the vitality of things would go down too. Not to mention the people reposting are kind of keeping the whole thing alive for their virtual points, and if they can't do it at scale by posting the same video in 20 different places anymore things might just stop. I feel like a lot of social networks are relying on a an orbiting cloud of garbage that just recirculates constantly. I'm surprised no one has started uploading, upscaling and posting America's Funniest Home Videos clips.

dang5• 4 years ago

Flagging from the new page doesn't go nowhere - it's handled the same way as flagging from any other page.

jfb2• 4 years ago

I remember learning that when Pixar did a pan-and-scan version of one of their movies, they wouldn't just use a floating 4:3 frame over the 16:9 original -- they'd re-render the movie, moving items around as necessary, to get a better framing. The perfectionism was palpable when we'd meet with them.

bee_rider• 4 years ago

This seems like a huge advantage with mostly-digital movies.

I wonder how easily a Marvel or Starwars movie could be re-rendered like this. Technically there are actors of course, but they are usually the main focus of the scene anyway...

gabereiser• 4 years ago

For green screen shots this would be fine. For VFX LED wall shots, this wouldn’t work. However, movies and shows today are shot in a high resolution that you can crop/edit into most formats.

Green screen shots lend itself to resizing better due to the ability to fill in more of the scene while pulling the subject matter away a bit to fit the format.

jonny_eh• 4 years ago

VFX shots are not pure CGI, it's a whole lot of layers composited on top of each other, think Photoshop but video. So making tweaks to any given shot would not be as easy.

hondo77• 4 years ago

I wonder how easily a Marvel or Starwars movie could be re-rendered like this.

Fortunately we don't have to worry about 4:3 anymore.

pj_mukh• 4 years ago

This is a huge selling point for rendering software in almost all media. Working on something for ecommerce here: https://www.dabble.so/

acuozzo• 4 years ago

Disney did this with "Lady and the Tramp". Cinemascope and an academy ratio versions were animated individually in order to make the best use of the frame shape.

jfb2• 4 years ago

I didn't know this. The level of effort involved there is just staggering.

navels• 4 years ago

Do people not search HN before posting?

locallost• 4 years ago

I remember the stories from Ed Catmull's book Creativity Inc. Although not that they tossed the movie out after almost deleting it anyway, like it's said in the article. The story about starting from scratch was that they left the team that did it alone, but it was their first movie and it fell flat. Reading about it then felt like a dick move and kind of questionable, but even more so later when I'd read that Lassiter, who was the most critical one, is a certified creep / jerk. So I don't know, but I guess if you're successful in the end your opinion is never questioned.

sdevonoes• 4 years ago

In order for her to work on the movie while out, they had plugged the machine up to the local network and copied the whole file tree over.

Now image the same situation in 2022:

- there is no acceptable backup for the whole movie (or let's say, app)

- only you know there is one acceptable copy... but it's on your personal laptop. You did copy it back in the day because, well, let's say you wanted to run the app in our local machine

There is no way you would tell your team/boss: "Hey guys, no worries! I have a perfect copy of it!", because why on earth did you copy work material into your personal laptop?! That's the rule number 1 of every modern tech company: don't mix work stuff with personal laptop (and the other way around: don't use work laptop for personal stuff).

asciimov2• 4 years ago

- If you can trust an employee with a $25K ($50K today) computer, trusting them with a complete source tree isn't a stretch.

- I'd be more pissed off that somebody didn't have a complete backup somewhere, even if it was a personal laptop they used for work.

- Back then, most people didn't have home computers. Hell nowadays a home pc isn't a guarantee, as most of my (non-engineering) coworkers only have cellphones and ipads at home. And while I agree don't mix work and play, most of these people do.

serf• 4 years ago

Back then, most people didn't have home computers.

you had a much different perspective of 99 than I did.

That factoid might have been true for-the-public-at-large, but certainly not for the offices of large CGI or dev firms.

Daub2• 4 years ago

The computer in question was a Silicon Graphics workstation (pictured in the article). What the image does not convey is the sheer gargantuan size of SG machines. Desktop computing it may have been, but in some cases people actually used their SG boxes as physical desktops.

The SG Octane that features in this story was actually one of the more petit versions, but still it would take two people to move comfortably.

One is shown here: https://www.youtube.com/watch?v=VHsA8iq4N0s

SG machines used to be the object of desire for all digital artists, sadly now gone the way of all things.

2muchcoffeeman• 4 years ago

Susman was the Supervising Technical Director and the machine was a Silicon Graphics workstation. Certainly sounds like she was given a work machine to do her job.

cnorthwood• 4 years ago

Early in my career I was working for a well-known anti-virus firm. They had a crash of their Perforce servers and it turns out the backups were not good and the most recent good one was several months old. They painstakingly restored it from the good backup and forensic recovery of the drives, but there were several files that were still missing to fully restore the archive. They did know the hashes of the files, so what they did was push out a bunch of custom virus signatures to the dogfooding (internal testing) environment which flagged any files with those hashes as viruses, and then asked anyone who had a hit to send the file into central IT to get it resolved. It was fairly clever I thought!

fortran77• 4 years ago

The real culprit in this story isn't the person who may have accidentally typed an "rm -rf" -- it's the person who was responsible for making backups.

StevePerkins• 4 years ago

Everyone talking about backups, and the need to avoid blame during disaster recovery. And that's all great.

But MY big takeaway is the final section. After all that, the studio threw away the movie anyway. Had their people make a different movie, working 100 hours per week for 9 months straight. That sounds like an absolutely dystopian hellscape of a culture, and the tone of the article seems to celebrate it. Marveling that those people had the "opportunity" to experience that level of "camaraderie".

My first couple of jobs out of college were pretty "death march"-ish. And yeah, maybe there's something to having an experience or two like that under your belt when you're young and single. But this Pixar example here is just straight up abusive and exploitive, and I'm not sure that a celebratory lens is how we should best view it.

WirelessGigabit• 4 years ago

Does someone have Red Alert 2 stored somewhere?

Pixar almost deleted Toy Story 2 (2012)

Related Stories