liamca

[Full disclosure, I work on the Azure Search team]

Cosmos DB has tight integration with Azure Search, to the point that you can choose to extend your Cosmos DB to Azure Search with a few clicks right from the portal to allow for full text search over this content.

You can learn more about how to do this at these two links: https://azure.microsoft.com/en-us/blog/adding-search-to-docu...

https://docs.microsoft.com/en-us/azure/search/search-howto-i...

Liam

@Avalaxy, I completely agree with you and I have also taken this set of comments and passed it on to the MSDN team.

[Full disclosure, I work at Microsoft] @an4rchy, I think if you open the page in private mode it will most likely ask you again. If that does not work, feel free to contact me and I can pass on your comments (my email is my HN alias + microsoft.com).

@Avalaxy, it would be great when you click Yes or No, that you also try to add a comment about your issue or why you liked the page. I can not speak for every group, but most groups I know of review the voting and comments for all of their pages really carefully. For example, my team has a weekly discussion on this and what we plan to do to improve the pages. If a page gets a negative vote along with a comment why it poor, it is much easier to improve the page, however, if it simply has a negative vote we have to make guesses as to why it was poor.

[Full disclosure, I work on a service called Azure Search]

Very nice site! Since your site is so much based around search, I thought I would pass on a few suggestions based on what I saw. If you happen to be using a search based engine for your content such as ElasticSearch, SOLR or maybe Azure Search :-), there are a few simple things you could add to make the experience a little smoother. Suggestions in the search box are nice to allow people to quickly see results as they type. You could even add thumbnails of the images in the type ahead such as you see using the Twitter Typeahead library (http://twitter.github.io/typeahead.js/). I also noticed that your search does not handle spelling mistakes or phonetic search (matching words that sound similar). Finally, through the use of Stemming, search engines can often help you find additional relevant content. For example, if the person is looking for mice, but your content has the word mouse in it, this will bring back a match. Since you don't have a lot of content, this can really help people find relevant content.

Hope that helps.

tkmcc is correct, the other big area that people typically want is the ability to easily tune their search results. For example, you might want to give more weighting (or boost it higher in the search results) to an item if the search text is found product name then if you find it is in the product description. Or perhaps boost a result based on the proximity (geo-location) of the item in relation to the user.

Full disclosure, I am a Program Manager on the Azure Search team.

arafalov,

The purpose of starting with this "Simple query syntax" was to try to keep things as simple and straightforward as possible for both the developer and the users of search. We have only exposed the query syntax that we have found that customers we have been working with so far have needed. I am sure there will be more, and as you say, since the core is ElasticSearch, if the demand for things such as configurable analysis is there, we can certainly look to expose it in our API. For things like this it would be great if you could cast a vote in our UserVoice page (http://feedback.azure.com/forums/263029-azure-search). By the way, this is a great place to go to see the feedback and suggestions from customers we have been working with so far.

Liam

tmarman,

It is agreed that the jump from $0 to $125 (preview pricing) is a large jump and I can say we are definitely considering something in the middle ground. I am curious, what types of features would you be willing to give up for a lower price point? For example, one option might be for us to consider using smaller VM's, but that would greatly reduce the document count (perhaps 1M docs max), QPM's (perhaps max 1 QPS) and/or possibly limit the ability to support for high availability.

Do these sound like reasonable things to give up for this lower price point? Do you have alternate ideas?

Liam

tracker1,

I hear you on this and to add another justification for the need for openness, let me give you another example. With ElasticSearch there are some amazing tools that are available. A few that come to mind are Kibana and LogStash. Given that we have our own API layer on top of ElasticSearch, we are not able to support these tools even though we are using ElasticSearch at our core. This is most unfortunate.

There are many reasons why we put an API layer on top of Azure Search. One key reason is that for our particular customers, we found that there were things we could do in our API that could simplify the interaction with Search. In fact, we have a system called Scoring Profiles that allow you to easily (I hope) set weights on important fields and attributes to quickly tune the results of your search based on what is important to you. No coding required. Another reason we don't just expose ElasticSearch is that ElasticSearch allows you to run random code. This is generally not a great idea in a PaaS service and can often lead to issues. There are a number of other reason that I'll skip for now.

We still need to do some thinking in this area. I hope we can get Azure Search to a point that we can enable tools such as Kibana and LogStash to work with our service without compromising the goals of what we are trying to build. Not only would this allow us to really open up the types of things people can do with the service, but I suspect it would really help reduce concern around vendor lock in. We'll see if we can get there...

Liam

ChuckMcM, you are absolutely right. Nailing down QPS rates are an incredibly tough thing. Not just for Azure Search but also for most Search engines that I am aware of. Things like #'s of facets, complexity of queries all play a part in what a search engine can serve up from a QPS rate. When we say ~15QPS we try to point out that this is based on an average index of the ones that we have seen from our typical customers. Certainly some customers may see way more QPS on a single search unit and others will see less.

The main markets (or scenarios) that we target with Azure Search are eCommerce Retail, User Generated Content sites (such as a recipe site or Hacker News) and internal organizational apps. The interesting thing about internal organizational apps is that we are seeing more an more users are finding that search is a natural way to navigate and explore their data. Users are typically far more knowledgeable of using search to explore their data thanks to engines like Google and Bing then then are with say SQL.

We actually don't have an official SLA yet for this preview. That is one of the goals of this public preview which is to really determine what we can realistically promise for our v1 release.

Yes, privacy is a thing for sure. It is interesting that you say lawyers because we have had a number of companies in the law field that have wanted to use Azure Search. Things like indexing of case documents is quite popular from what I have seen. In many of these examples (and also with Helathcare especially), privacy or more specifically encryption at rest as well as compliance (such as HIPPA) often become critical. As of today we don't have either. We don't have encryption at rest and we do not have HIPPA compliance for Azure Search. Of course, this will be a goal and I guess we need to start somewhere. The encryption as it relates to search is actually going to be a really hard thing to do properly so that will be an interesting thing for the future.

By the way, WikiPedia is one of the datasets we often test with our service. Feel free to ping me as we have a loader for the WikiPedia dataset that I could look into sharing with you if would you like to play with it and Azure Search. My email address is my YCombinator username + microsoft.com.

Liam

Yes, the core of Azure Search leverages ElasticSearch. It does however have an API layer on top of ElasticSearch.

Hi ChuckMcM,

I'm a Program Manager on the Azure Search team. I am going to correct your numbers a bit. Even though you can have a maximum number of 36 search units, the number of partitions you can create (currently) is 12. Partitions, by the way is what you increase to allow you to increase the number of documents. With this limit of 12 partitions, the maximum size of an index is actually 180M documents or 300 GB (not 900 GB as you stated). So far, we have found that the vast majority of customers we have been working with fit well below these limits and in fact even more of the majority fit into the 1 partition (15 M document / 25GB) range.

For a very few customers we have talked to, there is a need for more than this and for this we can actually allocate a much larger system that has much higher ranges. We have an azuresearch_contact email address on the pricing page (http://azure.microsoft.com/en-us/pricing/details/search/) with more details if you need this.

To your other question about racks and search units. You can think of a search unit as a dedicated Azure VM for your usage. For each additional Search unit you create is an additional VM for your use. Each VM has a certain amount of capacity that it can handle. If your needs grow beyond what you can get with a single search unit, you can move the dial up to increase it whether it is increasing replica count to add more QPS / High Availability or increasing partitions to add more documents / faster data ingestion. The way you calculate the number of search units you have is replicas x partitions, where each search unit (during public preview) is $125 US / month. By the way a single replica can handle about 15 QPS which for most customers is more than enough. But even with this, the ability to scale up and down is pretty important to a lot of people. Imagine Black Friday in the US where a retailer gets hammered with searches, yet only wants to allocate increased replicas for that day to handle the increased query load. There is a bit more information on this here: http://azure.microsoft.com/en-us/documentation/articles/sear...

Hope that helps, Liam

Hi blutoot,

I am a Program Manager for Azure Search, so you will have to take this response as perhaps being a little ones-sided. I think you will find that getting elastic search up and running in Azure to be very easy. As to your point about not worrying about scaling or management issues using ElaticSearch on Azure VM's, I don't agree with this. There are so many things that you will still need to be concerned about. For example, Azure will periodically do VM updates and patches which will cause your VM's to occasionally be restarted. This of course would affect your search availability which means if you can not permit downtime, you need to think about replicas across VM's and how to manage availability groups so that the patches are applied in a way that you can avoid downtime. Then you have to think about how you are going to handle partitions and how you will shard your index across machines in the case where you need more than one VM to accommodate the amount of data you have in your index. All of these things and parts that we in Azure Search take on for you and make easy for you to scale up/down whether you need more replicas for higher QPS or higher availability or more partitions to allow for greater numbers of documents or faster data ingestion. There is actually quite a lot more to managing a search service (even in a cloud hosted IaaS environment), but hopefully this gives you a few things to think about.

Liam

Hi MichaelGG,

I am a Program Manager for Azure Search and as curiousDog mentioned, yes along with DocumentDB, we also announced Azure Search which is a PaaS based full text search service. We actually leverage ElasticSearch at the core of this service and as chippy says about spatial search, we do have the ability to provide a pretty solid geo-spatial search capability thanks to Elastic Search and Lucene. To nemothekids's point, is it very unlikely that we will offer this as a local (non-hosted) version because we found that although ElasticSearch is awesome, one of the common complaints many admin's have is the complexity around managing systems such as ElasticSearch/SOLR/Lucene at high scale and how difficult it is to implement more advanced search capabilities such as tuning and relevancy. Those are areas we think we can add a lot of value being a fully managed service. Longer term, we think this will also allow us to bring even more value on top of search by adding in other Microsoft technologies. For example, we could tie in Bing Maps to allow you to easily tie in reverse geo-coding right into your search. Or perhaps allow you to leverage Bing's synonym list so that you could allow people to search yet find results that are synonyms to commonly searched words (i.e., user types in shoes, but in your content it is referred to as footwear). Multi-language support is actually one of the big things we want to tackle in the short term and we believe that the NLP from Office will really help jump start us with this.

Liam

Hi ryanburk,

I am a Program Manager for the Azure Search team. I am glad to hear you think our service looks promising. You are right about searching across indexes. This is something we heard often from a number of customers we worked with before today's annoucement. There are often ways of working around this, for example by merging content into a single index, but this is obviously not workable for everyone, so you are also correct, that this is really just a matter of time.