Amazon Elasticsearch Service

https://aws.amazon.com/blogs/aws/new-amazon-elasticsearch-service/

by johns • 11 years ago

147 46 11 years ago

aws.amazon.com

bdcravens2• 11 years ago

I wonder what this says about Amazon CloudSearch?

room271• 11 years ago

It says it is not very good. There are a lot of problems with Cloudsearch. By contrast, Elasticsearch is great and not too difficult to run yourself in AWS, or indeed elsewhere.

Aside from product differences, Cloudsearch was phenomenally expensive, particularly if you had a larger dataset but low read/write volume, as Cloudsearch did not use EBS (at least not in a way you could control), unlike RDS for example.

dubcanada• 11 years ago

Nothing? Amazon has lots of Amazon + non Amazon services. They know people use Elastic Search, so people will give them money. Therefore make it.

Jgrubb• 11 years ago

Well that's a tiny bit annoying, since I just set up our own cluster 3 days ago to centralize all our logging...

SEJeff• 11 years ago

For reference:

https://aphyr.com/posts/317-call-me-maybe-elasticsearch

And the updated:

https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-...

aaronkrolik• 11 years ago

Will amazon support the head plugin? Can we enable our own plugins?

sscarduzio• 11 years ago

They support just ICU and Kibana. Shield is not there too.

icpmacdo• 11 years ago

I thought there was already an AWS Elasticsearch product? Am I remembering something different or is this coming out of beta or something to that effect?

imaginenore• 11 years ago

You're probably thinking of CloudSearch

https://aws.amazon.com/cloudsearch/

ihsw2• 11 years ago

There is CloudSearch however it doesn't support schemaless stuff.

Looks like this is AWS' answer to that.

There's also the issue of CloudSearch taking in document batches rather than individual documents, which means you have to do a fair bit of more leg work to get your stuff in there. ElasticSearch can take in Logstash inputs which is handy and quite performant.

Plus a lot of data analysts are used to Kibana so there is some appeal there.

necubi3• 11 years ago

Seems very expensive compared to just running ES on instances yourself. Using i2.xlarges (as we do for our ES cluster) it's $1.194/hour for managed ES and just $0.853/hour for spot i2 instances. If you're will to reserve for a year, it's only $0.3555/hour.

Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version.

beachstartup• 11 years ago

Given how easy ES is to operate

you overestimate the average web dude's ability to run a service that actually works, and stays up.

you also underestimate the budget that lazy devops people have when their employer is used to (i.e. captive to) to a $100k amazon bill that really should be $25k somewhere else.

balls187• 11 years ago

Given how easy ES is to operate, I don't know why anybody would pay for a less flexible, more expensive hosted version.

Easy. Amazon's service is fully managed. Getting ES running on EC2 is easy enough. Making it fault tolerant, automated backups to S3 and multi-az capable, takes more work. Even more to make sure it works properly.

Add the time and complexity required to test your recovery/tolerance/redundancy plans, and you pretty much lose out on the savings.

Given the choice of spending 2-3 days getting everything configured right, vs letting Amazon deal with it, I'd pick Amazon.

Using platitudes: the time I save not dorking with infrastructure is time that I can spend solving my customers problems.

dangrossman3• 11 years ago

+1. This is the reason I pay for Amazon RDS/Aurora, where the price difference versus the same EC2 instances is even greater. Sure I could set up a database... and replication to another database... and script scheduled snapshots... and build monitoring and fault tolerance against all the ways things can break... but by the time I've set all that up, I've wasted a week or more building servers instead of building a business.

ihsw2• 11 years ago

Given the choice of spending 2-3 days getting everything configured right

It's never 2-3 days, especially when the next dude managing it needs to be on-boarded. He might not be familiar with how you do things.

This externalizes all that management overhead.

One less thing that you (and your co-workers, and the next dude after you leave) can break.

Also all the configuration settings are exposed via a neat UI rather than hidden in various files scattered out and about. Yes source control, Puppet, Ansible, Docker, etc makes this vastly simpler but then you need to hire and train people that are familiar with these things.

typpo• 11 years ago

The last time I set up an Elasticsearch instance it got owned a year later due to a remote code execution vulnerability [1]. I found out because Amazon asked me why I was DOSing people.

For people who don't have time or resources to actively manage an ES cluster, security is another reason to go with Amazon's service.

[1] http://bouk.co/blog/elasticsearch-rce/

curun1r• 11 years ago

Beyond that, it seems that they've spent the time to integrate ES with IAM, so having the ability to grant fine-grained access to ES using IAM policies will be a big plus for some people.

We're currently using Found to host our ES index and we had to come up with some pretty creative solutions to avoid embedding the auth credentials in our applications that need access to ES. With the new service, we'd simply setup the correct IAM policies for ES and assign the proper IAM roles to the instances running our application and be done.

jowiar• 11 years ago

That price difference comes in at under a half day of developer time per month. That's rounding error in the cost of shipping something.

necubi3• 11 years ago

That's the per-instance price, so multiply it by the number of nodes you need. We use about a dozen, so the difference for us would be around $7000/month.

adrr• 11 years ago

When it breaks, you can call/email someone to look at it. Factor in the cost of having someone on retainer or hiring a person that has experience with hosting ES.

23david• 11 years ago

Call and/or email Amazon support to fix ES issues?

necubi3• 11 years ago

Hahaha... that's not how AWS works (unless you pay them hundreds of thousands to millions of dollars a month). Amazon is terrifically unresponsive even for medium-to-large clients.

ssalat2• 11 years ago

This new aws service offer is really boring, I don't see a real advantage migrating to it anytime soon.

What will be the expectation here:

- Autoscaling functionalities out of the box

- Better speed than on regular EC2 instances (I/O, Network)

- More visiblity, better monitoring out of the box (e.g. Marvel)

- Lower pricing than running on plain EC2

Still a long way to go here...

dangrossman3• 11 years ago

RDS seems to be doing just fine with a large markup over EC2. Why would the managed version of a service be lower priced than the same infrastructure being managed?

ssalat2• 11 years ago

Amazon has more possiblities to run a really huge cluster and balance the different load peaks of the search nodes. The whole business model of AWS is based on this... but you're right as well, price isn't everything and comfort worth money. And yes, I love RDS as well, especially Aurora... it's a great deal!

davidzyc• 11 years ago

That's soo true.

fideloper• 11 years ago

Opinions on the price being high seem to miss the point of AWS.

AWS is very expensive relative to other clouds, with exceptions so some very nice services (S3, SQS, SES, Route53).

"Core" (Very Important™) services such as EC2/RDS are (appropriately) expensive due to the tooling and high availability options available. I'd expect the tradeoff for their hosted ES to be price, since I'm buying availability and less headache, not just the search service.

I know for sure that I don't know enough to monitor JVM applications. I'd love to pay for their ES service if/when it made sense for a business.

spankalee• 11 years ago

Will Elasticsearch ban queries for AppleTV and Chromecast?

phpnode• 11 years ago

Solr and Sphinx

bitroliest• 11 years ago

We will definitely be checking this out. Our experience with another vendor and ES has not been great. From having the whole cluster come down when we're doing batch inserts for initial loads to having 10s timeouts on random queries during low activity windows with the vendor saying "dunno, everything looks good here" (when we measured that this 10s were spent waiting for ES).

evboyle2• 11 years ago

If you're looking for a search backend without having to worry about managing infrastructure, you should check out Azure Search. I'm a dev on the product. If you're interested in trying it out let me know. You can ping me at my username @microsoft.com if you have questions or need pointers.

nullrouted2• 11 years ago

Side question: Do you know if Azure will every allow traceroute through their network?

Thaxll• 11 years ago

It seems weird to handle the number of instances on top of AWS service, I mean it's AWS they should provide a solution where you don't have to think about that...

It looks like ES installed by Puppet on top of EC2.

ShaneOG• 11 years ago

sa-east-1 gets the short straw again.

The service does not support any i2 and r3 instance types in the sa-east-1 region.

odiroot• 11 years ago

Nice, they even support Kibana.

Just few days after I managed to finally set up ELK stack correctly by hand.

andrewvc• 11 years ago

Interesting, This would be Amazon's version of Found ( https://www.elastic.co/found ), which is Elastic's (makers of elasticsearch) aws based cloud solution. I wonder what this means for cloud search?

I should note that I work for Elastic on the Logstash team and used to work on Found.

johnhkg• 11 years ago

How would this compare to Algolia ? Does anyone have experience with elasticsearch and algolia to shed some light on this ?

Amazon Elasticsearch Service

Related Stories