ArchiveTeam rescues Justin.tv videos

https://twitter.com/textfiles/status/476064989879349248

by sp332 • 12 years ago

101 25 12 years ago

sp3322• 12 years ago

I guess this answers why they're not doing archives anymore. 1,000 TB of videos with fewer than 10 views!

kcorbitt• 12 years ago

Yeah. Do we know what Justin.tv's hosting costs for all that data was, or what infrastructure they used to store and serve it? That's a lot of data to take care of for a service with no direct monetization.

Dobbs2• 12 years ago

It is really out of date but the high scalability article has some information http://highscalability.com/blog/2010/3/16/justintvs-live-vid...

baddox• 12 years ago

Where do you see that statistic?

Dobbs2• 12 years ago

The rest of the tweets in the thread:

What about grabbing the 9 views, 8 views, etc? We have more time!

-@erazmus

The problem is that at some point, we end up with a LOT of disk space used for VERY obscure videos.
Normally, archive team does not care, but in this case, the disk space will increase to over 1.1pb. PETABYTE
That's space that can be used to save a LOT of other work out there that needs love. Tough decision.

- @textfiles

dbpatterson• 12 years ago

They said they archived 10TB (and that covered everything with at least 10 views), and that the total including all videos would be 1.1PB, ie 1100 TB. So there is over 1000TB of videos with less than 10 views.

joshu• 12 years ago

In the posted thread.

ANTSANTS• 12 years ago

Read the rest of the conversation:

  >What about grabbing the 9 views, 8 views, etc? We have more time!
  >The problem is that at some point, we end up with a LOT of disk space used for VERY obscure videos.
  >Normally, archive team does not care, but in this case, the disk space will increase to over 1.1pb. PETABYTE
  >That's space that can be used to save a LOT of other work out there that needs love. Tough decision.

timr• 12 years ago

I don't think anyone has mentioned this yet, so I might as well: unless they changed it after I left, JTV has been deleting videos older than a week for many years. I wrote the original (well, almost original) system, and IIRC, we took it down to a week of storage back in mid-2009 or so.

JTV caught quite a lot of static for "not giving more than a week's notice", but it's pretty hard to do that when you're going from one week of storage to no weeks of storage.

Also, since people are asking: back when I first worked on it (circa late 2008), we started by installing (again, IIRC) 40TB of RAID6 disk space, and that was enough to store about a month of video before we ran out. By the time I left, we had something like 4-5x that capacity, and we were down to a week of archive storage. So that should give you some perspective on the amount of data involved....and it's probably gone up substantially since then.

stingraycharles• 12 years ago

JTV caught quite a lot of static for "not giving more than a week's notice", but it's pretty hard to do that when you're going from one week of storage to no weeks of storage.

I have trouble understanding this point you make. What prevented them from announcing something along the line of "We keep videos for one week at the moment; in 3 months time, however, we will stop this and there will be no more storage." ?

ohhoe• 12 years ago

I'll be glad if they remove a bunch of the early stuff, to be honest. I was in the closed beta with iJustine and a few other people, and people were dicks to us whenever we got on camera.

textfiles2• 12 years ago

Since people are interested... a fact that may be obscured from the conversation and statistics is that somrthing like 550gb of those videos have ZERO views.

Kudos to archive team!

AustinDizzy• 12 years ago

How can videos with at least 10 views also have zero views? The ArchiveTeam is only keeping videos with at least 10 views because the total size of those videos with less than 10 views is 1.01 Petabyte.

textfiles2• 12 years ago

Sorry for not being clear. Out of the 1.1 petabyte, something like 50-60% of that has zero views. Archive Team duplicated everything with 10-infinity views, which was about 10tb.

duskwuff2• 12 years ago

550 GB, is that all? Or do you mean 550 TB?

userbinator2• 12 years ago

The page for their project: http://archiveteam.org/index.php?title=Justin.tv

I'd be curious to know how many hours (days? months? years?) of video content JTV actually has.

kd5bjo• 12 years ago

5 years ago, it was 22 hours recorded per minute[1]. That translates to 25 years per week, which is their previous retention term. The volume has probably gone up substantially in the intervening time.

[1] http://mashable.com/2009/05/21/justin-tv-usage-stats/

userbinator2• 12 years ago

So they're huge numbers, but still pretty small in comparison to YouTube which claims to have 100 hours of video uploaded every minute [1] - meaning they probably have at least several millenia of stored video... wow.

[1] http://www.youtube.com/yt/press/statistics.html

aroman• 12 years ago

I love the ArchiveTeam.

Jason Scott, the leader of the project, gave a fantastically entertaining talk about how they saved Geocities, Yahoo! Video, and Friendster — using a Distributed Preservation of Service Attack. Definitely worth a watch.

https://www.youtube.com/watch?v=-2ZTmuX3cog

ArchiveTeam rescues Justin.tv videos

Related Stories