New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SSD-Backed Storage

http://aws.typepad.com/aws/2012/07/new-high-io-ec2-instance-type-hi14xlarge.html

by jeffbarr • 14 years ago

196 102 14 years ago

aws.typepad.com

beedogs• 14 years ago

Amazon will find some way to make this slower than shit and less reliable than a campaign promise.

jedberg6• 14 years ago

I've used them and they are quite performant. Definitely lives up to its promise.

oomkiller2• 14 years ago

Hopefully we will start to see some other providers offer SSD-backed storage since Amazon does it now. It would be nice if they offered it on some smaller instances too though.

thegyppo2• 14 years ago

Storm on Demand have done SSD storage for a while now, pretty solid IOPS numbers too. I'll run a benchmark comparison tonight.

thegyppo2• 14 years ago

Ran some benchmarks, really insane IOPS on this plan. Pretty average CPU performance (they're using fairly old E5620's).

http://blog.serverbear.com/post/27553311076/hi1-4xlarge-benc...

j-kidd• 14 years ago

Check out this long thread on WHT, started 2 months ago:

http://www.webhostingtalk.com/showthread.php?t=1151090

From my limited testing, a 1 Node instance from GridVirt at $30/month can compile gcc-4.6.3 within 15 minutes (not exactly an IO-intensive example...)

ehsanu1• 14 years ago

This has been a long time coming, but AWS has consistently been improving their service (as long as you can ignore the particularly bad reliability as of late).

It's telling that they have only enabled this for a huge (quadruple extra large) instance type. It's probably hard to make this work for someone who just wants a 10GB disk with great IO. The problem at the low end is that disks are larger and would thus have to be divided up to make proper use of them, leading to IO contention..

The high IO options will probably only ever be available for pretty large instances.

oomkiller2• 14 years ago

It seems like they could spread a bunch of smaller instances over an array of SSDs, maybe just offer less space at a higher price? SSDs are naturally better at concurrent access, so while you might not get 150k+ IOPS, you could get at least 10k or so, with the expected low random access times. Imagine an SSD-backed EBS.

cperciva4• 14 years ago

It's telling that they have only enabled this for a huge (quadruple extra large) instance type.

My guess (not based on any knowledge of EC2 internals) is that they don't have any way to do fair I/O sharing between guests. If they did, they could split these boxes into 32 small instances with 1 ECU, 1.7 GB RAM, and a 60 GB disk with 2500 random reads / 250-4000 random writes per second.

XERQ6• 14 years ago

Xen offers easy ways of doing fair I/O sharing between guests. These servers they're using are most likely multi-tenant systems with 256-512GB of RAM and 6-12TB of SSD storage. Providers don't like keeping expensive systems around that aren't making money, especially when demand changes every hour, so I expect that they have at least 4 instances sharing the I/O of each host (especially when they mention broad ranges of expected I/O).

The most likely reason for not slicing these systems up to smaller instances is they want to maintain consistent, high performance I/O.

bravura2• 14 years ago

What are good use-cases for on-demand high I/O servers?

At $3.10/hr, these instances work out to $2k/mo. There are probably many more cost-effective options if you want a 2TB SSD server.

Since the benefit of using EC2 is that you can provision instances elastically, what are the sorts of scenarios in which one needs to provision high I/O servers elastically?

[edit: A few minutes of Googling, and I can't find any dedicated servers with 2 TB of SSD.]

saurik4• 14 years ago

That is only one of the benefits of EC2. If you are not using elasticity, then you have to factor in the reserved instance pricing, which drops the prices down by 71% (as in, to 29% of the list price; and I mean even if you include the up-front cost: that's overall savings). You, like most people who comment on the price of EC2, do not seem to be taking this into consideration. :(

joe_bloggs• 14 years ago

What are some of the other benefits, other than elasticity or on-demand ness, and not having to spend a huge amount upfront?

jcdavis2• 14 years ago

"Since the benefit of using EC2 is that you can provision instances elastically" That is certainly a benefit, but I think many would argue that its far from the only benefit.

And as with all EC2 pricing, reservations drop the price substantially. A 1 lease reservation for a "heavy utilization" instance comes out to $7280 per year + $0.621 per hour for a total of $12719

surrealize• 14 years ago

> $7280 per year or $0.621 per hour.

I believe that's $7280/year plus $0.621 per hour, or about a thousand dollars a month amortized over a year.

jedberg6• 14 years ago

The one that comes to mind is that you want to do a map/reduce job and need a large and fast working space to do the reductions.

Otherwise, as noted below, the most likely use case is in a reserved situation running Cassandra or Postgres or something.

cperciva4• 14 years ago

What sort of map-reduce space needs fast random I/O? The only things which come to mind are disguised table joins, where the right solution is to use sorting instead.

(Not trying to be argumentative, I'm just trying to figure out what you have in mind.)

tiernano• 14 years ago

closest i can find is Hetzner's EX8S[1] which is EUR99 for the box itself without disks, and another about EUR80 for 4 240Gb SSDs... This is a little less than HALF of what Amazon give, but costs about EUR180 a month and EUR150 for setup... but you do loose the elasticity of Amazon... Mind you, it is 1/10th of the price...

[1] http://www.hetzner.de/en/hosting/produkte_rootserver/ex8s

bigiain• 14 years ago

I cant think of an example, but I'd guess this'd work out as a good cost-effective solution for people who need to do "big data" map-reduce type jobs, but with less utilisation than 1 week/month. Or, if you can solve the "getting the data in" problem (perhaps you've already got all your data stored on S3), you could use one of these for an hour or two at a time, perhaps running end of week or end of day batch processing, at a somewhat lower cost that having a similar sort of pay-by-the-month colo server.

XERQ6• 14 years ago

[edit: A few minutes of Googling, and I can't find any dedicated servers with 2 TB of SSD.]

I am the founder of SSD Nodes, Inc. (http://www.ssdnodes.com) and we offer lower-sized SSD-backed storage in addition to custom cloud and dedicated plans that range from 1-12TB of local SSD storage, at comparable performance. [/plug]

corin_2• 14 years ago

Given your "1-12tb" range doesn't list prices online, can you tell us whether your prices are comparable? At $1249/month for 2x 200GB SSD, it seems unlikely, but maybe I'm wrong?

rurounijones• 14 years ago

On-demand high I/O servers...

Say I have a batch process that has huge I/O requirements that has to run once per month and be finished within X hours of starting or SLAs are broken. (Plenty of these types of custom workloads exist in the enterprise)

I can either buy a server with specialist enterprise-grade SSD / Fusion-IO / whatever (> $20,000 most likely) for this once a month process or I can spin up one of these high I/O servers for 1 day per month for a grand total of $50.

In this scenario, this new server type is a godsend.

mbreese2• 14 years ago

I do genome mapping where our indexes won't entirely fit in memory. It would be very handy to be able to spin up a few of these instances, load the indexes from an EBS volume onto the local SSDs, then run for a couple of hours or so. This is a very I/O intensive job that we need to run about once a week, but then the rest of the time could be idle.

SSDs would make our jobs run significantly faster. So much so that we've toyed with the idea of adding SSDs to our in-house cluster, but couldn't quite justify the costs. This might actually shift the cost savings to get our lab to migrate to EC2 as opposed to our in-house or university cluster.

Gmo3• 14 years ago

We are also facing the same kind of problems in my company, regarding genome assembly and mapping.

That's definitely something we will look into :)

revorad• 14 years ago

I'm working on a data visualisation app, which is getting a lot of interest from biologists and bioinformaticians. I'd like to learn a bit more about your work. Can I email you somewhere? Or please drop me an email at hrishi@prettygraph.com. Thanks!

lonnyk• 14 years ago

Any chance we could get an example of the data set and the calculation that needs to be done?

zurn• 14 years ago

How long would it take to read the 2 TB from EBS?

arethuza2• 14 years ago

Monthly financial consolidation for large companies? My employer has a rack of servers for one application that are used very heavily for a few days a month then are hardly used for the rest of the time - the databases are currently being moved to internal Fusion-IO devices rather than Fibre Channel SAN drives.

Spooky23• 14 years ago

The key consideration there is how many consulting hours will you need to figure out how to build up and tear down whatever Oracle/SAP/etc software that you need to do your financial calculations on the EC2 VM.

You may find that Fusion-IO is cheaper than $50 + 3 months of consulting time.

yummyfajitas• 14 years ago

A reserved instance is about $650/month. One possible use: run your master postgres on it and use streaming replication to transmit the data to something more durable.

(Don't process cc transactions this way.)

_pdeschen• 14 years ago

We use ec2 within our telecom infrastructure which manages peaks of several thousands of calls with full duplex call recording on (think IO here) We are using the elasticity of ec2 to scale on demand.

This kind of instance is a nice addition for us as far as our auto scaling is concerned.

earl• 14 years ago

on-demand doesn't matter.

If lots of your infrastructure is in ec2, you may need a good db server inside ec2 that is used constantly. eg if you have a read heavy cassandra workload, you need ssds.

damian2000• 14 years ago

The pricing in the blog post is a bit unclear - the prices on http://aws.amazon.com/ec2/pricing/ are ... US East $3.10 for linux and $3.58 for windows. EU West $3.41 for linux and $3.58 for windows.

(Reserved instance prices are cheaper)

jbarham• 14 years ago

FWIW it costs $27,156 a year on-demand or $12,720 as a reserved heavy utilization instance.

For heavy analytics workloads I'd bet that Google BigQuery (https://developers.google.com/bigquery/) would be cheaper and faster and more reliable.

amnigos• 14 years ago

This would be very useful for elastic EMR workloads, will be good for killing I/O bottlenecks.

Fizzer• 14 years ago

This is a game changer for big sites on EC2. The key word here is local: 2 TB of local SSD-backed storage.

In this video [1], Foursquare says the biggest problem they're facing with EC2 is consistency in I/O performance. They say that the instance storage simply isn't fast enough for them, and while EBS is fast enough when RAIDed, it isn't consistent since it isn't local (EBS is traffic goes over the network). Reddit has also complained about EBS, but they've been able to move onto the instance storage.

If you're willing to reserve the instance for 3 years, the average monthly cost becomes only $656. That's quite a good deal.

Foursquare says in that video they're planning to migrate off of EC2, in part due to I/O performance. I'll be interested to hear whether or not this instance type changes their minds.

[1] http://www.10gen.com/presentations/MongoNYC-2012/MongoDB-at-...

jedberg6• 14 years ago

Netflix got a huge performance boost for Cassandra using the SSD instances:

http://news.ycombinator.com/item?id=4264888

XERQ6• 14 years ago

If you're willing to reserve the instance for 3 years, the average monthly cost becomes only $656. That's quite a good deal.

The only problem with reserving that instance for 3 years is that better hardware always comes along, especially with the cost of SSDs coming down significantly every year. Usually if you're in the big-data space, your hardware is likely retired after 24 months (12 months if you're well funded) so locking yourself in for 36 months might be a bad investment.

jaylevitt2• 14 years ago

Has Amazon ever bumped the specs on existing hardware types? Or do they just create new hardware types? e.g. is it possible that if you get a 3-year reservation for an h1.xlarge, by 2015 h1.xlarge might have newer specs?

I had thought that EC2 reservations were upgradeable, but a quick check on the forums shows you're right, they're not. Of course, you can play your own "tiered usage" game, like laptops in IT departments, where the old h1.xlarge becomes cheap enough to use as a second-tier machine and you go reserve the h1.xxlarge for Cassandra.

velominati• 14 years ago

Don't reserve for 3 years.

AWS cuts their costs at a relatively reliable rate. We've done the match an found that the 1 year reservations are absolutely worth while, but that the 3 year reservations are not. Granted that was for our specific workload / use case.

Money has a time value, and this stuff is getting cheaper fast.

btb• 14 years ago

Would these be suitable to run a single big SQL Server on? I mean specwise they seem perfect for our size/use. They say data will survive a reboot, but what are the chances I would some day have to wake up in the middle of the night and having to restore the database to a new server? Would some of the recent amazon downtimes be one of those cases where that could happen?

jedberg6• 14 years ago

That would be a really bad idea unless you have real-time replication to another server. SSD or not, you don't want to write data to the ephemeral store without a real-time backup, unless you're willing to lose it all.

Loic3• 14 years ago

Another approach is something like OVH SSD servers (24GB ECC memory, 2x300GB SSD, £210/month) https://www.ovh.co.uk/dedicated_servers/mg_ssd_max.xml

If you are using MongoDB, you take 3 or 4 of them and shard and you backup with "conventional" storage for the replica set. You end up with a 6 node cluster for less than the price of this Amazon instance.

Lesson: You need to have a business which can benefit from a lot of start/stop of your instances for them to make sense from a pure financial point of view.

cbg02• 14 years ago

Some notes:

1. You can't order from OVH if you're from outside their list of approved countries. 2. You're not using any RAID and those are desktop grade SSD drives, and they tend to die out, sometimes without a clear warning, as they're not really intended for 24/7 server use.

Loic3• 14 years ago

1. The list of countries from where you can order servers is expanding on a regular basis. If your country is not there yet, it should come in the future.

2. You have 2 300GB with a RAID card (battery powered). So, you can put in RAID. For the reliability, I keep my fingers crossed (I have some of these servers) but no failures yet and an interview with the operators said that basically they an extremely good reliability. This is not marketing in this case, this is because they need it to be financially sustainable.

By the way, do you know what kind of SSD (SLC, MLC, real disks or cards?) are used by Amazon?

mrb• 14 years ago

Anyone can report on the model of SSD they use (via ATA IDENTIFY)?

My guess based on perf characteristics is each instance has 2 x 960GB OCZ Talos 2 C Series SSDs: http://www.oczenterprise.com/ssd-products/talos-2-c-sas-2.5-...

spartango6• 14 years ago

It's very likely that they are using Intel SSDs (likely 320/330, maybe 510+) for a variety of reasons:

Intel has a good drive reliability history, and is very enterprise friendly in bulk purchasing. Intel has had excellent firmware for their drives, which at the datacenter scale is valuable--people who have dealt with raid controller firmware(including Amazon) know all about this.

Traditionally, Amazon has not used SAS drives in EC2, opting for lower cost SATA drives. It's also unlikely that Amazon is using small numbers of high capacity (>=500GB) drives because they still aren't perfectly price effective; price per gb is ok, but replacing a failed drive is more costly.

Also keep in mind that to get to today, Amazon has been rolling these drives in huge numbers out through two enormous data centers, so it's unlikely that Amazon has picked a brand new drive (say the latest OCZ Vertex 4).

There are other factors that Amazon has bumped into while testing drives, but they remain unreported and internal.

rit• 14 years ago

It seems we all may be missing the "backed" part - which I did on my first read through. They don't seem to be revealing how much of the logical volume is actual SSD, which I think is why they're instead putting down IOPS numbers.

Still, a huge and significant improvement over anything previously available. I'm looking forward to playing with it.

spartango6• 14 years ago

It's very likely that the entire volume is SSD, otherwise the tp99 would be atrocious (and if you look at the netflix numbers its actually quite good). The reason for the broad ranges is more likely that the SSDs have some inherent performance variability due to wear leveling and GC processes. Additionally, as time goes on, multi-tenancy will start to come into play (if Amazon has smaller SSD instances or larger machines), which will stay within that range comfortably.

jedberg6• 14 years ago

The whole volume is SSD.

josephcp• 14 years ago

Reminder since it's a pair of SSDs and most people will probably look into using this for their DB store: If you use current generation controllers/software & SSDs, you're going to have a bad time if you turn on RAID and don't know exactly what you're doing.

TRIM ( https://en.wikipedia.org/wiki/TRIM ) isn't supported with RAID on SSD today on hardware controllers and most distributions of linux don't support TRIM on RAID out of the box if you're doing software RAID, so you're going to see performance plummet like a rock after you do one pass of writes on the disk. In many RAID configurations, you're going to zero-write the entire disk when formatting it, so performance is going to suck from the get-go. For this reason, even if you have a tiny database and don't expect to write 1TB worth of data, your performance might still suck. Personally, I haven't tried linux software md TRIM in production, the patch is pretty recent, so you're on your own here (if possible, scaling out horizontally may be a solution to consider for redundancy, I have no idea what Amazon using for SSDs, but recent Sandforce generations fail all the time, so plan for that).

If you don't know to look for this issue, you're going to be scratching your head when your RAID10 SSD configuration write throughput is worse than a single 7200rpm drive. On the other hand, IOPS on SSDs are AMAZING for databases/datastores. Amazon may have solved this for you already behind their visualization instance, and they might be running their own software striping behind whatever raid you're doing, so be sure to test it out fully first.

nirvdrum2• 14 years ago

Interesting. Do you happen to know if using LVM instead of mdadm is susceptible to the same issues?

js2• 14 years ago

In addition, keep in mind that MD (software raid) does not support discards. In contrast, the logical volume manager (LVM) and the device-mapper (DM) targets that LVM uses do support discards. The only DM targets that do not support discards are dm-snapshot, dm-crypt, and dm-raid45. Discard support for the dm-mirror was added in Red Hat Enterprise Linux 6.1.

Red Hat also warns that software RAID levels 1, 4, 5, and 6 are not recommended for use on SSDs. During the initialization stage of these RAID levels, some RAID management utilities (such as mdadm) write to all of the blocks on the storage device to ensure that checksums operate properly. This will cause the performance of the SSD to degrade quickly.

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6...

spartango6• 14 years ago

Don't assume that its actually a pair of SSDs under the hood. It's very likely that its actually ~8 SSDs md/LVM'd together as two volumes(JBOD, no raid). I'm pretty sure they've taken steps to ensure that TRIM is working, as they'd burn through drives/complaints too quickly otherwise.

That said, you're absolutely right about being cautious/not RAIDing the volumes. There are almost no RAID configurations that support TRIM at present, so it's definitely not a good idea to be RAIDing up these drives. Just go JBOD.

benmccann• 14 years ago

What's a good way to utilize the two disks for your datastore? Run mongodb for example one disk and backup to the other? Run one sharded instance on one disk and another sharded instance on the other disk?

tlack• 14 years ago

With MySQL (and Oracle before that) it was common to simply move different parts of the database data files to different disks. I don't use Mongo so I can't speak to that, but the concept works pretty much universally. See here for more information about spreading your database around multiple disks: http://www.mysqlperformanceblog.com/2010/12/25/spreading-ibd...

jaredstenquist• 14 years ago

Since I'm spending more than $1,000/month for an RDS master, which is backed by EBS, i'm intrigued at the idea of running our database off of these. Of course I'd lose all the awesome automated features of RDS, but worth considering.

Maybe AWS will release a High Performance RDS option that runs off of them. Wishful thinking.

drudru11• 14 years ago

This is nice, but they need a smaller instance that costs less that also has SSD. Until then, I'm not going to blow $5k a month on a system like this.

level09• 14 years ago

wow, would like to do a redis benchmark on one of those !

New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SSD-Backed Storage

Related Stories