Ask HN: Buying servers?

https://news.ycombinator.com/item?id=379471

by slackerIII • 18 years ago

26 35 18 years ago

news.ycombinator.com

I might be buying 10-20 servers in the next few months, and I'd really appreciate any advice folks have about getting the best deals. I'm mainly looking for some fairly reliable 1U boxes with 8 cores and 16-32 GB RAM. Disk is not that much of a concern.

Can I get significantly better prices if I start talking to a sales guy vs going over the web for places like Dell and HP? Are there any white box vendors you guys would recommend? Has anybody learned any lessons or had a great experience with a vendor they can share?

vaksel• 18 years ago

go on eBay, lots of refurbished stuff there, a guy I know buys all his stuff from one user.

Me personally, I prefer to just lease them

mrtron• 18 years ago

Leasing computer hardware instead of buying results in tax credits in Canada, so many people take this route.

brk• 18 years ago

Yes, you will get better deals by talking to a sales person, ESPECIALLY in about 2 weeks (as the quarter and year are coming to a close and people are trying to make their numbers).

Make it clear that you know the value of what you are after and are ready to make a purchase as soon as you find the right deal.

johnm2• 18 years ago

(A) Lease vs. Buy: If you can swing it, lease them instead - tax benefits, etc.

(B) Do you really need 8 cores per box? I.e., you can get used dual-dual machines a lot cheaper than 50% of the price of the dual-quad machines.

(C) If you know what you want and are buying new then going with a white box builder is significantly cheaper than buying from Dell/HP/etc. I've been pretty happy with ASA especially since they've delivered/picked-up/serviced machines directly to/from/in our colo but there are plenty of decent vendors and they most end up using SuperMicro-based systems anyways.

(D) Do you really need 1U? Why? Power/cooling issues are putting more constraints on things than size at many places and a 2U is often cheaper, more flexible, fixable, etc.

(E) Really knowing exactly what you want can make a big difference in talking with the salespeople. Buying at the end of the quarter is a good thing if you can swing it.

(F) More ram is more better and 32GB is the current sweet spot in terms of lots of memory for a good price. Pay particular attention to the motherboards that you're considering as there can be a price premium at the various steps depending on if you need the flexibility to upgrade the memory a lot later.

(G) You mention that disk isn't much of a concern. Is that because you just want lots of compute nodes or because you're putting everything into a NAS/SAN or what?

slackerIII2• 18 years ago

I was thinking 1U because I don't need a lot of disk and it gives me more options about how many I can put in each cage. What kind of power limitations have you run into?

As for the disk, I just don't have a lot to store.

philjr• 18 years ago

Normally there is a power threshold measured in Kilowatts or amps on your cabinet. Amps is generally easier to work with. 16A or 32A would be pretty standard. Really, you need to ask your colo facility. Be careful with a 32 amp A & B feed if you're using them for redundancy. Ideally you don't want to overload the A or B feed if one of them is down.

The advice given to you regarding Dell servers is pretty good. They also have a power estimator on their website which you can download which give really good indicators as to what your power usage might be.

Basically, the chances are you wouldn't be able to pack 42 1u servers into a 42U cab and power is becoming more and more of an issue.

Extra note: newer generation dell servers are much less hungry than the old ones. The PE 2950's are great 2U machines, easy to work with and I'd vote for them over the equivalent 1U form factor 1950s from my experience.

johnm2• 18 years ago

If you don't need to cram a lot of boxes into a tight space then 2U will usually give you better airflow and cooling.

Also, when those on-motherboard NICs die, you can more easily/cheaply add additional NIC cards to the 2U boxes.

In terms of power, that's actually the main limitation in most colo's these days. Some colos are down at 15 amps per rack, many seem to be in the 20 amps per rack, and some you can get up to 30 but you will pay for that one way or another.

Even at 30 amps per rack, for the kind of machines that you're talking about you are still likely to be less than about 20 machines per rack. I just spec'd a build out of the same horsepower & ram but with more juice for e.g., FC cards to a SAN and the safe limit was 16 machines in a 30 amps per rack colo and 8 machines in a 15 amps per rack colo.

Shopping around for power information can be tricky because some of the colos like to play marketing games to get you to think there price is better than others but the power charges can eat you alive if you need more than what they allocate in their base charges.

iigs• 18 years ago

We have a smattering of 100-150 HP DL380s and BL460c blades. We've seen approximately a 5% infant mortality rate (DOA or issues in the first 90 days). Depending on the channel HP's custom configurations come as a base and add on accessories -- if you order a dual 5160 DL380 with 32G of ram and 4x146g SAS you might get a preassembled 1x5160 4G 1x146 and a CPU, 14 2g DIMMS, and 3 disks, each in a separate shipping carton. You then spend about one person day per two computers unpacking individual components from cardboard and shipping foam boxes and installing CPUs, DIMMs, and disks. It's not my favorite thing to do, and it means that you only get about three computers per pallet. It's also, of course, not burnt-in beforehand, so you get to rack and memtest them for a day or two yourself.

In contrast we've had really good luck with a regional whitebox VAR -- they would assemble Intel reference platforms and Supermicro systems, burn them in, and ship them, assembled, in the one box the case came in. As far as I know we've had closer to a 1% failure rate over three years with these (exclusive of HD failures but not environmental, over-heat, issues).

On balance it's a tough call, because HP (and I assume Dell) have a much slicker lights-out-management solution than the last IPMI demo I saw. Maybe the whiteboxes have improved, but I assume the HP machines are still easier to support.

yellowbkpk• 18 years ago

Another HP story. Not nearly on the scale you're talking about, but the company I works for uses HP DL360 G5's as the hardware they sell to customers. Before we ship to customers, we open the box and install software and such. As a tester/engineer for this software, I've never seen a dead DL360 arrive. Yes, the shipping containers are huge. HP's iLO2 software (integrated lights-out) is excellent.

Also, whatever machine you get, make sure you buy some extra rails and hardware. Store the extras somewhere and label them with the date purchased and which systems they are for.

volida• 18 years ago

You may consider 8anet.com. You can customize the server according to your needs. They also setup for you the server for 20 or 50 dollars. I don't remember exactl

davidu• 18 years ago

I've bought over 500 servers from 8anet over the last six years. I'd highly recommend them. Call them and talk to Mary Z.

brianm• 18 years ago

So, some lessons learned...

1. lights out management matters

2. ensure your OS is happy on the hardware ahead of time

3. if you are expecting to make regular purchases, with growth for example, tell the vendor that

4. always negotiate

5. dell actually makes good servers now

6. look hard at the power density you get wherever your servers will live and factor cost of rack space into server cost numbers -- if you have low power density, slightly pricier servers with lower power reqs may win out

blender• 18 years ago

For whitebox vendor you might want to look at Supermicro

Cheers

kvogt• 18 years ago

We've tried several vendors at Justin.tv, but we always end up circling back to Dell. Pricing is great, spare parts are readily available on Ebay, and they're quality machines. Definitely get in touch will their sales guys and plan your purchase for the end of the month, or better yet, end of the quarter.

You're making the right choice on 1U 8-core boxes. We've done the math and there's no better combination of price, performance, density, and power consumption out there.

-Kyle

jbyers• 18 years ago

Note that Dell's end-of-quarter is the end of January, not December. http://finance.yahoo.com/q/ks?s=DELL

e1ven• 18 years ago

I'll add another vote for Dells-

a) Their servers rack very nicely, particularly when you're using Dell racks. No tools are necessary, which saves a Lot of time when you're setting up/moving the machines.

b) The management tools are much better than you get from SuperMicro/offbranded machines- You can generally find a hardware fault quickly from their boot CDs, rather than diagnosing manually.

c) As Kyle mentioned, the parts are ubiquitous- Dell still carries parts for most of their machines, but even if you can't find it from them, you can find it from another vendor.

d) Because Dells are so common, it's easy to find other's experiences with a software/hardware combination. http://en.community.dell.com/forums/

e) Support. Don't underestimate this when choosing what vendor to buy from.. With one of out offbrand vendors, a replacement drive took at least 2 weeks to be shipped out and arrive.. If you pay for the Dell premium support ($200ish per server), they'll drive a tech to the spot within hours.

They're less than ideal on storage pricing- It's often cheaper to buy drives elsewhere.. Further, their live reporting tools (Check HD status, etc), aren't as good as they really should be.

I'd certainly call up a sales rep and have a chat with them. They'll usually give you a discount if you're buying any quantity- If they hesitate, ask if they'll throw in free gold support. They'll normally do that to save a sale, since it doesn't cost the rep very much.

-Colin

jwilliams• 18 years ago

> You're making the right choice on 1U 8-core boxes.

If it's appropriate... I've found Apple XServe is a competitive 1U 8-core as well.

lsc5• 18 years ago

the biggest problem with dell

is only 8 ram slots. if you want 32GB ram, this means 4GB modules... you will end up paying more than a grand more per server than you would if you got a box with 16 ram slots.

wehriam2• 18 years ago

Amazon EC2 has a large VM if you don't want to purchase or lease hardware yet:

"Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform"

$0.80/hour - about $730/month.

cconstantine• 18 years ago

I have very little experience with EC2 (or managing server farms). I have worked at Amazon and been responsible for an internal service there that ran on a server farm, but we didn't use EC2 and another department managed the machines.

Has EC2 been reliable enough lately? I've heard horror stories of 8+ hour downtime multiple times a year.

lsc5• 18 years ago

if you leave that on all the time, and plan on keeping it for more than a month, that's a horrible deal.

I can build a super micro server w/ 32G ram and 8 cores for around $2500, and I get to keep it. Heck, I'll rent you one of mine and host it for $512/month.

Of course, if you only need it during the day, or temporarily or something, ec2 is an awesome deal.

wehriam2• 18 years ago

I won't dispute that in many situations owning machines is the most economical way to go. But keep in mind:

* EC2 servers can be deployed quickly and cheaply - you don't have to overspend on hardware to meet an upper bound of predicted demand. The original poster needed between 10 and 20 machines - if he only needs 10 but purchases 20 he wastes $25k.

* Co-location fees are included

* Features like elastic IPs, elastic block storage, and security groups make managing machines easier.

merrick33• 18 years ago

I used to do IT Consulting and was responsible for procuring hardware from dell at a few companies including now my own.

What I learned - End of Quarter yields huge savings at Dell. If you are going to buy 10 servers for sure, create an account, spec out one server and call in to the small business division. Ask the salesman what he can do if you buy 5 of the one in your cart, he will give you a number let him know you have to talk to your partner put him on hold or call back. Then ask what he can do if you buy 10 - watch the price go down again on the quote.

mmmurf2• 18 years ago

You can save a lot of money by going with the sales guy. Just be sure he knows that you are growing and you'll be placing a big order every few months.

First go to the Dell website and price out the equipment. Then when you're speaking to the sales guy, tell him you priced it out on the website and it came out to about 88% of the actual price the site gave you.

Then tell him you've heard that talking to the sales guy is the way to start a business relationship and that you were hoping he could quote it out for you. Tell him you're ready to make the purchase decision immediately but the pricing has to make sense. Tell him the pricing on the website looks like it's marked up quite a bit.

It may take him a day or more to generate the quote.

Be aware that he has tremendous room to wiggle on some items, but not on others. Last order I placed the salesman could discount heavily some items but not others. You don't really care as long as the overall price makes sense. So when you eventually go over the quote with him, be sure to focus on your "budget" for the equipment, and not on where the specific pricing comes from.

You should also have a simultaneous dialog like this going with a rep from any other companies you'd buy from.

To the salesperson, the equation is effort vs time. If he knows you're ready to buy soon, then he has every incentive to make you a good deal b/c in a day the sale will be made and so he'll be willing to take a small commission in exchange for a quick sale. He has some constraints about how he can price items, but has a lot of flexibility on price.

If you're buying 10 to 20 servers you should be looking at least a 25% discount off of the website price. The only reason they publish those prices is to give the salesperson a starting point to work down from (or to sucker naive shoppers into paying full price via the web).

You want to come across as decisive and ready to buy at all times, if only he can solve the "issues with the pricing".

Frankly Dell's sales process stinks, and it is annoying to have to work with some of the salespeople there. But I've saved a substantial amount of money over the web prices by negotiating with them, so it's worth it.

You may also be able to save money by modifying some of the configurations slightly, b/c Dell lets salespeople discount based on the need to move inventory.

Don't be afraid to tell the salesperson it's OK if he needs to ask permission from his manager to make the deal, but that you're eager to get the order placed today. Tell him he can put you on hold while he speaks to the manager.

Good luck!

mmmurf2• 18 years ago

Oh forgot one thing. You should expect to get free shipping on an order that size. When it gets to the last negotiation then you can say "Ok, I'll do it if you can get me free 2 day shipping on it".

slackerIII2• 18 years ago

Great information -- thanks!

ezmobius• 18 years ago

Supermicro servers have served me very well. All of our clusters at engineyard.com are built on supermicro 1u's with 32gig ram and 8cores. Highly recommended.

Freaky• 18 years ago

What's their LOM like? I can't say I've been impressed with cheapo IPMI cards which (attempt to) piggyback off the on-board ethernet.

lsc5• 18 years ago

yeah, the IPMI kinda sucks... more than DRAC, even (which is worse than HP ILO)

I use SuperMicros but I just redirect console over serial (they can do this even for the BIOS setup, nearly all 'server' boards can) then I hook the serial to another server with a $5 USB to rs232 dongle (or to a terminal server in the case of racks with a lot of servers)

then you want a remote rebooter. You want that even with DRAC, 'cause DRAC isn't particularly reliable.

bbhoss-synsol• 18 years ago

I like Sun servers, they can be expensive, but you can get some good deals if you are accepted into SSE (Sun Startup Essentials). For me one of the best things about them is the ILOM, which is badass as far as I'm concerned. Also, their N1 System manager lets you manage and provision a bunch of servers from a single location. It will also keep track of the environment on all of them, but thats nothing you couldn't build with SNMP traps etc. Also, I must mention that if you pay for the support, it is awesome, I had a problem with system drives on a Friday, and they overnighted it and I had it Saturday. They also don't charge you unless you don't send the defective part back, with THEIR free packaging and shipping. It was also nice of them to include 3-4 3M anti-static wrist straps with my shipment of new hardware :)

rmason• 18 years ago

I can second the Supermicro solution. Worked for a hosting company in the past and we never had a Super Micro machine fail. They cost less and are built a lot beefier than comparable Dell's.

carl_2• 18 years ago

Lots of great comments so not much add/kinda like others have said:

When dealing direct with vendors (Dell, Sun and IBM from experience):

1) Price you're given is not only based on current purchase, but future purchase. Telling your sales rep 'I need X servers now, but I expect to need a further X per quarter growing to X after X time' then they'll fight harder for your business. Basically, present your business as a growth opportunity to the vendor.

2) When speaking to your first sales rep, ask at what price/order level they generally deal in. The reps who handle bigger clients have access to bigger discounts.

3) Don't under or over-specify your machines without calculating your later options. Buying an extra processor for one machine now will be a hell of a lot cheaper/easier than buying the matching processor in 3/6/12 months. In the same respect, not buying that extra processor now might be more prudent because a new chassis with the latest and greatest processor in 6/12/24 months might be a better option. (should be part of your scalability predictions anyway).

4) Know your stuff.

4.1) We all know reps bs, by knowing what you're talking about you can reduce this tedium.

4.2) If you ask which processor is better, then it'll always be the more expensive one (or more importantly, the one with the better margin for them which is not always the most expensive).

4.3) mmmurf said it, but to re-iterate. Time = Money. If a rep is holding your hand too much, then you won't get as good deals.

5) Know your vendor and their enemies. 5.1) When is there end of quarter? 5.2) How have they done compared to comparable vendors over the last quarter, year, etc? 5.3) Which vendors are pushing X line/type of servers at the moment? 5.4) Have comparison quotes.

6) Specify a date when you will be looking to purchase. 5.1 has a MASSIVE impact on this.

carl_2• 18 years ago

Also, Dell price-match RAM to crucial for me which might be useful for the servers you need.

lsc5• 18 years ago

Hah. I know this one. Ok. first, are you willing to use a wrist strap, to buy an anti-static mat and verify you are grounded?

http://prgmr.com/~lsc/luke_opterons.jpg (my setup)

If you are not, stop right now and take someone else's advice about how to get a good deal out of dell and HP (me, I can't afford dell and hp prices... my business model cant take that kind of markup on the ram. Dell and HP have OK deals on the base server, but you get screwed if you fill it with ram.)

Ok, you have the wrist strap and the dissapative mat? Ok, now first the ram. If you want 32GB ram, remember that 4GB modules cost twice as much per gigabyte as 2gb modules. this means you really want a board with 16 ram slots.

Second, the hardest part of assembling a computer is fitting the motherboard into the case and designing the cooling. Me, I buy SuperMicro SuperServers where that sort of thing is alrealdy done. It probably costs me $200 extra over some homebrew solution, but it saves me time and gives me a better finished product.

Here is the computer I settled on:

http://supermicro.com/Aplus/system/1U/1021/AS-1021TM-T+.cfm

Now, just search around for the lowest prices. if you are buying 20, you can call salesjerks, but my experience has been that they spend a lot of your time (time that I can bill out for close to $100/hr) and almost never give you a better deal on parts (remember, the salesguy wants to get paid, too.)

I got the aformentioned case from newegg for around $1200.

Next, do the same with CPUs. I ended up buying quad-core opteron HE chips at 1.9ghz for $260 or so each on newegg again.

Now, for ram, I very rarely beat the price on shop.kingston.com. They have a nice thing for finding the ram that fits your motherboard, and a good warranty.

the 2 servers in that photo? they take 1u of rackspace, around 350 wats power, and the whole ball of wax, for 2 servers, each with 8 cores and 32GB ram and 2x1TB disks, came in at under $5000.

But again, if you are unwilling to use anti-static equipment, buy the servers from a reputable vendor, and never open them.

Ask HN: Buying servers?

Related Stories