ASK HN: Is there room for a new search engine?

https://news.ycombinator.com/item?id=3060681

by jim_shook • 15 years ago

9 7 15 years ago

news.ycombinator.com

I'm relatively new in posting to the HN community, though I read it a bunch, and consistently find the discussion on here to be the most intelligent in tech. I figured it would be worth posting a concept I've been thinking about to hopefully get some good feedback. Not sure if its the right forum for something like that, but here goes.

To begin, I do search engine marketing for Roomorama.com, I'm part of the founding team, and love working there, though am always thinking about new potential projects, like I'm sure most of us do.

The theory I'm working under for the concept I'd like to get feedback on, is that "search", as it currently exists for the most part, is an indexing and parsing of HTML, while the web, and what people actually want when using search, isn't HTML. Its objects that match their long tail criteria.

When you think about it, the results that come from the current process, are only a good approximation of what searchers really want. If I'm looking for a blog post about the American Jobs Act from a liberal blogger under 500 words, will Google be able to really help? Or if I'm looking for an event, the weekend of October 15, in Denver, that's good for kids, will Google help by returning HTML pages? Not really. If you're a savvy web user, you go to the vertical search engine that is best (indeed for jobs for example, or eventful for events, or whatever). But that requires the person knowing those engines, which most people in the real world dont.

Google is trying to get structured data into search results (ITA software acquisition as example), but it has been kind of a mess. Try searching "apple pie recipe" and look at the mess that they offer up. I have no doubt theyre going to improve, but the major search engines architecture and infrastructure is so fundamentally based on the concept of HTML web pages being the backbone of the internet, that I think its going to be hard for them to really move away from that.

Is there room for a search engine that starts from the beginning, looks at each vertical, brings structured data into the system (via partnerships with the vertical search engines that exist) and intelligently returns the results that their searchers really want, with a smart way of being able to sort and refine?

Just throwing it out there. Appreciate any thoughts you'd have!

27182818284• 15 years ago

I think the thing the worst thing Google does right now is its search. I challenged myself to use Duck Duck Go (DDG) for a month and after a month I'm still using it. The only thing I miss about Google's search are the occasional fun Google logos on holidays. But that's it! There is no real difference. I still find the content I'm looking for, etc.

What that tells me is that yes, there is great room for new search engines. I hear whispers about a super elite team at Google that does amazing algorithm work and data analysis work to keep their engine going, but at the same time I'm getting comparable results with Bing, DDG, etc. Several times people have posted a link to blind search where you see the results without the branding and it becomes difficult to discern the difference.

jim_shook• 15 years ago

Yeah I think Blekko had a search engine monte thing where you could run a search blindly across 3 engines and see which ones were best. At least for me, no search engine came up on top.

The theory I'm working with is that its because they're all looking at the web as a series of web pages, instead of discrete objects or data.

So like a search for "2 bedroom apartment in new york with a doorman" awkwardly brings me results of web pages (that are SEO-ed to the hilt :). Instead, don't I just want a series of apartments that match my criteria?

And couldn't that data be fed from those offering this structured data in the first place such as Roomorama.com (the site I've been working on), or Airbnb or others?

Thanks for the reply :)

dewiz• 15 years ago

you are talking about semantic search. it's a whole """new""" world, which requires different things to be in place. G+ and FB are walking on that path, linking different things to eventually offer a smart search engine. not to mention lifelogging which is part of all of this, since people needs to put a meaning on their stuff, differently from blog posts where there is too much information without clear structure.

md1515• 15 years ago

There is always room for a new search engine. I think Paul Graham wrote an article that questioned "do you think 100 years from now we'll be using the same search engine? I don't even think Google believes that".

There is a lot of room for innovation - there always is. Also, a lot of niche search engines are popping up. I briefly perused TC this morning and saw an engine focusing on family heritage...

Pheter• 15 years ago

Yes. My ideal search engine would return answers rather than a list of links to answers.

Brain dump of ideas follows:

Search for: "Best cheesecake recipe?" Result: Ingredients, step-by-step guide, perhaps a video too.

The search engine would adapt to my knowledge and, if I do not know much about baking, and/or cheesecakes, it would inform me that cheesecake's can be broadly categorised as baked or unbaked. It would recommend the recipe for one type to me based on which is easier for a beginner to bake, but with the option of overriding the default.

Search for: "Best steak"

Now this is a little ambiguous, so I would be prompted for further information such as "price range, distance, the time that I would like to eat at".

Result: x restaurant, which is y distance from you, serves the best steak at z pounds.

If applicable the search engine would inform me that if I am willing to travel half a mile further then I am able to get an even better steak for half the price!

It would also be great if I could make a reservation from the same page displaying the results.

----------

Now I realise that the above is pretty much impossible for a variety of reasons, but I would love to be able to pay for something like this.

ActVen• 15 years ago

Yes. I don't have many specific complaints with Google Search. However, as companies get larger they often look "upward" at upper management and investors instead of looking "downward" at their customers. What I mean by this is they are often more concerned with pleasing their bosses, meeting revenue goals for investors, and generally not rocking the boat. This presents a great opportunity for small companies to disrupt the market.

You should definitely read "Seeing What's Next" by Clayton Christensen to get a better idea of where the weaknesses can be found in companies that dominate the market. A focus on semantic search certainly sounds like it could be a good area for a new entrant to gain some ground. Good luck!

sandroyong• 15 years ago

I believe there is.

Let’s examine what we have so far. When you search for 'apple' in the library, where do you go? In truth, without indexing or the help of the librarian, you go roving through the library and gather all material related to or with the words ‘apple’ and ‘display’ your results. Google uses algorithms to do just that - and according to the materials’ the relevance and/or ‘hits’, it will be listed as such. And, when advertisers pay$ for their ads to be listed first - these listings will show up in the uppermost sections of the first page - if someone paid me at the library to look for certain material and display them first, I would too. Alas, this is the nature of things and we as end-users often have to muddle through the endless list of ‘stuff’ to find what were searching for. Yes, yes, we could narrow our search and type 'apple, fruit' and get more or less what we wanted but there is still ad ‘stuff’ that we had to dredge through.

My point is that it would be nice to have and indexing of stuff or a ‘librarian’ to go to to be able to help us narrow our search parameters. Suppose the network infrastructure ‘knew’ exactly where ‘stuff’ is - it’s more akin to you finding stuff in your room or house - if things are cluttered, you will be using algorithms to search through all that stuff; but if things were organized (since you organized it and put it there, and putting the algorithms back in your pocket), you wouldn’t have trouble finding it. Technically, this would mean that the information would be stored locally by each of the end-users on the network, and because it’s stored by end-users, the network 'publishes it' and knows where to 'look'. Just my thoughts...

ASK HN: Is there room for a new search engine?

Related Stories