Geocities Cage at the Exodus Datacenter

http://www.detritus.org/mike/gc/

by imaginator • 10 years ago

189 84 10 years ago

stpe• 10 years ago

Not sure what's the best find, the data center article or the author's boy band-esque photo: http://www.detritus.org/mike/pics/inpark.jpg

staked• 10 years ago

I love seeing stuff like this. Though I'd be a happy man if I never had to see the words "Veritas Volume Manager" again. Definitely one of my least favorite pieces of software from that era.

I worked for a company around this same time that had servers in the NJ Exodus data center. Used to have to head up there once a month to swap out tapes in the Sun L11000.

tyingq4• 10 years ago

Brought back some memories for me as well. In corporate IT, unix sysadmins had to know how to manage multiple different hardware architectures and operating systems that all did things different. AIX, Solaris, HPUX, Ultrix, etc, all with different filesystems, raid hw/sw, command paths, and so on.

That picture of the ethernet hornet's nest too. Ugh. At least it wasn't AUI cabling.

setq5• 10 years ago

Indeed. AUI was a killer. We had a policy of turning it into 10base2 right away with an adaptor, then 10baseT when that became a thing.

I liked this era so much I used to skip dive all the kit that was chucked out. Had myself a nice stacked Sun 1000E as a desktop in 1999, until I got the electricity bill. Must have cost as much as a house when it was new.

Then I found HP/UX was horrid. Had a run in with some HP N-class systems with Oracle. Yeuch, and that turned me to open source.

roganp2• 10 years ago

Veritas was definitely better than the alternative. I was a dba at the time and it was a godsend.

acveilleux8• 10 years ago

I still can't believe a top-tier UNIX vendor required an expensive 3rd party software to have halfway decent file-system. Or in the case of HP-UX, partitioning.

I get that Veritas also did logical volume, raid and so on and that was a guenuine value added at the time. But so many box I saw had veritas so they could partition and get a file system that was OK with 36GB+ monster SCSI disks.

RamenJunkie_• 10 years ago

And then years later you could download all of Geocities as a few hundred GB Torrent.

hogrammer• 10 years ago

Actually, it's only about 20% of geocities. Much of it has been lost

biofox• 10 years ago

Which makes me wonder: what would it take to match the capabilities of this setup with modern hardware?

setq5• 10 years ago

Probably just a couple of HP DL380 G8's with SSD arrays on Linux would run rings around all that kit.

Or a smallish EC2 instance and the whole of the rest in S3...

freeone3000• 10 years ago

Could be done with a moderate-spec single server. Replace the JBOD enclosures with on-server 8xSamsung EVOs in hardware RAID6, and gain a bit in speed because we don't have drives that slow anymore. Xeon E5v4 could replace the computational power. Network multihoming would be done the same, except with 10000baseT instead of multiple cables. Run nginx on it and you're good.

Or, one could host the entire thing on S3 with CloudFront at a fraction of the cost.

matthew1471• 10 years ago

I offered free hosting on my site for a little while, the legal issues of what people would do with free storage started to be the biggest problem. We're in a different era now.

Overtonwindow2• 10 years ago

Does anyone know of an article like this about the dial in modem systems and infrastructure for some of the early dialup services?

alex_hitchins2• 10 years ago

I worked for a large UK organisation that had cut back on their dialup access but still had 50 or so modems in racks for field workers to dial in to to get their email etc. Also some site to site links remained a dialup connection.

Sorry, no fancy pictures or anything interesting, just reminded me of the time.

nixgeek2• 10 years ago

At least a couple of the Dyn datacenters had dialup for out-of-band access back in 2013, think its all Ethernet now.

acveilleux8• 10 years ago

A few decent things I could find online that jive with what I remember:

https://www.patton.com/technotes/build_yourself_an_isp.pdf

http://www.gwi.net/behind-the-scenes-of-a-90s-internet-start...

https://news.ycombinator.com/item?id=8352432

The absolute minimum, and representative of the very first dial-up ISPs: http://www.linuxjournal.com/article/2025

http://www.datamation.com/erp/article.php/615281/My-own-priv...

There used to be a lot of nice material on this subject but a lot of it has been obsolete and rotted away from the web over the last 15-20 years. The larger dial-up ISPs used Cisco AS series boxes (or equivalent) with PRI (i.e.: phone over T1) connections (24 lines/each) to a centralized RADIUS server for authentication. They are/were the last hold outs providing dial up.

Smaller ISPs were more of a '94 to '99 thing. Usually they used cyclades or equivalent serial port cards with up to 16 serial ports per card and an external modem per port. Eventually this morphed into boxes with multiple modems in them and access servers that did the ppp termination and the authentication (to a RADIUS server) as scale increased. US Robotics was probably the best reputed player in the modem space.

halbritt• 10 years ago

Lots of people used the US Robotics Total Control boxes in the early 90s.

http://www.kmj.com/tcont.html

jsjohnst7• 10 years ago

In my smaller regional ISP we started off using USR courier and sportster modems with Cyclades cards before moving over to Livingston Port masters. I ended up selling the company before moving to all digital incoming lines (needed to support 56k modems).

_wmd• 10 years ago

I can't speak for early early dial up infrastructure, but by the turn of the century the equipment involved was not all that impressive to look at.

A typical configuration would terminate the analogue lines over ISDN, which supported somewhere up to 30 B channels ('voice calls') over a single cable, running alongside one or two Ethernet cables to a single router with one or more modem option cards installed.

We had Cisco boxes in the last place I worked that handled dialup, looking pretty much like this: http://www.ciscomax.com/datasheets/3600/Cisco_3640.jpg

aidenn02• 10 years ago

Are you sure you're not thinking of T-1 (24 B channels) rather than ISDN which was most typically 1 or 2 B channels?

[edit]

ISDN is a catch-all term tat included BRI (2 B channels) and T-1 (24 B channels) and E-1 (30 B channels) so the parent comment is correct.

jsjohnst7• 10 years ago

I'll dig up photos of the ISP I founded back in 1993. I've got the evolution of it from 1993 till I sold it in 97, but they were all film photos for obvious reasons. Surprised I never took the time to post before.

hacktothefuture• 10 years ago

I would love to see those!

Overtonwindow2• 10 years ago

Yes please !!

icedchai2• 10 years ago

I worked at a couple of small ISPs in the mid 90's, and built the dialup infrastructure of one from scratch. I remember using Xylogics Annex terminal servers connected to either 16 or 32 individual USR modems. There were stacks of modems, each with its own power brick, serial, and RJ-11 connection.

This was a few years before everything went digital with PRI lines (T1s that let you do digital modems, basically.)

acveilleux8• 10 years ago

So much memories from that era. I saw my first 1TB storage array back then, it was an EMC Clariion fibre channel raid box, it cost nearly 1m$ and it was a full rack of 9GB dual-ported 15k RPM disks.

roganp2• 10 years ago

And that was EMC's "low-end" array. The Symetrix (Symetrics?) I remember being much more expensive for the same throughput.

acveilleux8• 10 years ago

Symmetrix, two Ms. They were more of a mainframe thing I thought. I never really got to play much with the really big gear, Starfires, Superdome and so on. By the time I got to use a 64-way SMP box (Origin 3800) it was already obsolescent and replaced a few years later with a pair of dual-quad core Xeons.

vidarh5• 10 years ago

We had an IBM ESS "Shark" back then. A fantastic piece of kit but totally overkill for us in the end. It was basically the size of two or three standard racks, full of shelves for drives, and two RS/6000 AIX servers, and trays of redundant controllers.

You could pull trays of drives, one of the RS/6000's, or any of the controllers, and it'd keep on humming along thanks to redundancy pretty much everywhere. If any components went bad it'd call home to IBM via a modem.

And it could do things like automatic mirroring to a remote site.

1.5TB was a common configuration, but you could connect multiple boxes to increase capacity.

joezydeco• 10 years ago

"As you can imagine (and see from these pictures), this equates to a whole bunch of ethernet cables. Cable management gets increasingly difficult to grasp each time a new box is added to the mix"

Eeeeeek. I'm not in IT and I got the sinking stomach feeling when looking at this.

Thankfully there is https://www.reddit.com/r/cableporn/ to cure that.

tyingq4• 10 years ago

That mess wasn't common in any environment I worked at during the same time period. Pretty sure that would have got someone sacked in the various datacenters I happen to have worked in.

GFischer• 10 years ago

They did eventually fix it, see the newer racks:

http://www.detritus.org/mike/gc/rearpatchpanels.jpg

acveilleux8• 10 years ago

It was common in many customer cages. I even saw cages setup in 2000 where a cage contained 500k of sun and cisco hardware still in their box after 8 months. I guess that startup was growing so fast they hadn't gotten around to plugging things in yet /s.

More likely they either ran out of funding or never got the growth they expected. Back then there was often a large disconnect between H/w bought and H/w needed/used. But money was there and hypergrowth was "just around the corner."

dkresge2• 10 years ago

I seem to recall the adjacent cage hosting a Very Large Company's hot web-based email product which was cabled "organically" and made this look like art. (Hi Mike!)

sulam• 10 years ago

And right there is why Sun was such a hit in the 90s and crushed in the bust.

acveilleux8• 10 years ago

Everyone in SV in 1999 used Sun boxes. Racking Ultra-2 Pizza Boxes was being very Internet Professional and money was flowing freely enough to pay for them.

lostboys67• 10 years ago

yes in BT we where sun 4u servers at $40k a pop

jrnichols• 10 years ago

Yep. I was at Netscape in Mountain View around that time, and the data center was mostly full of SGI stuff and Sun stuff. The new datacenter on Ellis opened up, the AOL thing happened, and even more Sun hardware started showing up. E4500s, E450s, you name it. I had an Ultra 5 sitting in my cube just because I could. And then the Intel stuff started showing up in droves. Compaq servers running Linux. And I don't remember seeing a single Sun box show up in that data center after that...

jgalt212• 10 years ago

yes, and now computers are cheap and devs are expensive. The money has to flow somewhere.

chrissnell• 10 years ago

Great memories. During this era, I was at Citysearch and spent a lot of time in the Orange County Exodus DC. We had a nice rack of Sun gear, Extreme Networks switches and Alteon AceDirector load balancers.

I remember vividly that there was a cage down the row from us that was populated entirely by eMachines, which were a low end desktop PC that you could buy at Circuit City and Best Buy. We laughed at their cage but the company, 911gifts.com, ended up getting acquired for a nice sum, while our site and company was basically gone a few years later.

alexhawdon• 10 years ago

That's a great story. It'd be interesting to know whether that was all they could afford, or if they were being very forward-thinking in using COTS hardware and a failure-tolerant architecture.

beachstartup• 10 years ago

probably a combination of both.

i worked at a startup in the late 90's and the price of sun microsystems gear was mind-blowingly high. from memory: ultra 5 workstation with a scsi card and disk array was $15,000+. an ultra e450 server was on the order of $100,000 and went up from there depending on how you wanted to build.

of course, that was exactly the time people started switching to linux on x86 en masse. pentium pro's were good and cheap enough to scale out less expensively on a per unit basis. today you can buy a 72-core xeon server with gigabytes of memory and terabytes of ssd's and 4x 10G ethernet for less than 10 grand. amortized over its functional lifetime, it costs less than a cell phone bill.

scurvy• 10 years ago

Their revenue mattered a lot more than their COGS. At least in this case.

jsjohnst7• 10 years ago

I had forgot that GeoCities started migrating to NetApp filers after the Yahoo! acquisition. As many thousands (tens of thousands?) of the filers they bought from NetApp, Yahoo! really shouldve bought them when they had the chance I feel.

shams93• 10 years ago

I worked for geocities as a front end person back then. We got to work on some cool javascript stuff for the page builder app that was really pretty far ahead for its time. I got into trouble for bringing a laptop with suse linux on it into the office, back then linux was considered a major security risk, you notice all the machines are from Sun. They were also very early adopters of server side java, but most of the plumbing was composed of c code, the javascript people were kept far away from the server side and we didn't make it up to yahoo, but they did snatch up almost all of the unix admins.

dkresge2• 10 years ago

Wow, Page builder (formerly Geobuilder) -- crazy times. I don't remember the Suse incident! But your mention of server side java reminds me that we also experimented with server side javascript via Netscape Enterprise Server. Compared to the C CGIs that comprised most of Geo's backend at that point it seemed a panacea. Too many bugs and not ready for prime time later we stuck w/ Apache (but moved more to handlers).

tenaciousJk• 10 years ago

Good memories.

Does anyone remember where an Exodus colo facility was in SF around that same time (1999-2001)? That was my first visit to the area and I can't seem to locate the neighborhood now that I live here.

chiph2• 10 years ago

We were hosted at Exodus in the Herndon Virginia area around this same timeframe. The staff gave us a small tour and showed us a cage about the size of a double-wide trailer. Inside were two Sun Enterprise 10000 servers and a whole wall of drive arrays. Plus networking gear, tape drives, etc. Easily $7-8 million worth of stuff.

They said it was from a search engine we probably had heard of. Our guess was it was Altavista.

We didn't own our Compaq servers - we leased them from Exodus, like a lot of firms. And when the bubble popped, all those startups stopped paying for all that expensive equipment (which was now used and worth much less), and Exodus was on the hook as the owner of it all. Killed them.

Edit: Found this image from someone who picked one up for a song to add to their collection. From a prized million-dollar enterprise-class server, to being hauled around in the back of a pickup truck.

http://imgur.com/a/lXvOk

drewg1232• 10 years ago

Hmmm.. I'm pretty sure AltaVista ran on DEC Alpha hardware, not Sun. And Google used self-built high-density racks of Intel hardware.

Maybe it was Inktomi? I'm pretty sure they used Sun hardware.

chiph2• 10 years ago

I don't think we knew about Inktomi at the time, but it certainly was a possibility. Our other guess was Ask Jeeves, since we didn't think that Excite had the money for that kind of hardware. But who knows, maybe Exodus floated the bill for that too.

rsync• 10 years ago

"Hmmm.. I'm pretty sure AltaVista ran on DEC Alpha hardware, not Sun."

Yes, they did. Remember - the original URL for altavista was: altavista.digital.com.

nokahn• 10 years ago

I think all colos say crap like that. During a tour at switch, they pointed to a large cage and said, "It's a search engine that you all have heard about but I cannot name."

A week later, in Atlanta at QTS metro, they said almost the exact same words.

grkvlt• 10 years ago

If it really was a large search engine, then I imagine they would be hosted at multiple locations, so not surprising that they were mentioned at both places?

Geocities Cage at the Exodus Datacenter

Related Stories