One company's plan to build a search engine Google can't beat

https://www.protocol.com/neeva-search

by prostoalex • 6 years ago

264 321 6 years ago

magedqwani• 6 years ago

it subscribe model is OK for a starter but what are the search algorithms employed should produce open source technology if its want vast adoption

omarchowdhury2• 6 years ago

They use Bing API.

8organicbits2• 6 years ago

putting search results back at the top of search results

Watching Google slowly fill more and more of the search results with ads, this is an obvious and very welcome idea.

imustbeevil2• 6 years ago

Why watch? uBlock origin [1], the most recommended browser extension in the history of the internet, blocks the ads on Google. You can also right click and block any element you don't want in your search results, like "Top Stories" and "Videos". You're still getting the search results, if there's content in your browser you don't want to see, you have more control over that than Google does.

I haven't seen an advertisement in 10+ years. I don't really understand why anyone chooses to see them when they don't have to.

And sorry if this comes off as confrontational, I just see so many people talking about advertisements and it's difficult to have to tell each individual that adblocking extensions have existed for close to 15 years [2]. I wish there was some better way to spread this information so no one would have to see ads or comment about their existence ever again. The internet is so much better without them.

[1] https://en.wikipedia.org/wiki/UBlock_Origin

[2] https://en.wikipedia.org/wiki/Adblock_Plus

chickenthirty3• 6 years ago

uBlock Origin is great but doesn't work on Safari, which a lot of people use since Chrome is an incredible battery hog and Firefox is significantly slower on Mac.

None of the choices of content blockers in Safari successfully block the majority of ads on the internet.

Furthermore, it's not just about the surfaced ads. Even if you use uBlockOrigin, search engines like Google optimize for ad clicking, which will affect the search result ranking even if you have ads blocked. As a result, search quality has been steadily decreasing over the past decade (there have been hundreds of highly ranked HN discussions on this in the past).

Finally, uBlockOrigin is an amazing tool developed by 1 person. There is always the chance that, in the future, there are developments in browsers or ad-serving technologies that render it obsolete (e.g if Google decides to make a breaking change to the Chrome Extension API, like Safari did). In that case, it would be worthwhile to have alternatives.

freeopinion2• 6 years ago

I see two fundamental issues with this approach.

The first is punative. While it does, kind of, punish bad behavior, it doesn't go far enough. You may deprive Google of some small revenue, but you're still giving them a lot that contributes to their bottom line. They can still claim a googol of clicks and eyeballs, etc. People will still see them as the end-all-be-all, and voluntarily submit their sites to google's attention--ignoring any other options.

The second is the failure to reward. It is not enough to kill off bad actors if you haven't nurtured good actors to take their place.

In this case, the good actor could even be Google. If there are efforts to do things differently, we should find the ones we like and reward them. They will benefit and their competition will observe and imitate.

8organicbits2• 6 years ago

sorry if this comes off as confrontational

It didn't to me. I use ublock and take it a step further to use other search engines. Personally, I only notice how ad filled Google search is when I use someone else's computer.

alexhaber2• 6 years ago

Related: https://hammer.lol/

ZephyrBlu• 6 years ago

Is this a joke?

alexhaber2• 6 years ago

Probably

BbzzbB• 6 years ago

Looks like it's by George Hotz/Geohut, so half-joke half-serious perhaps?

atlgator• 6 years ago

It is a joke, sort of. George Hotz of comma.ai wants someone to start a Google alternative and charge a monthly fee instead of invading user privacy. He made the hammer.lol page on a live stream last week.

freeopinion2• 6 years ago

"If you want to replace google, send an email to this gmail address."

No, I think George is completely serious.

ve55• 6 years ago

Even if they make a superior product, how will they get users to use it anywhere near the scale they use Google?

devmunchies2• 6 years ago

They don’t need google scale, they just need profitability.

A premium search engine can have a place with professionals. Maybe they could sell enterprise licenses.

omarchowdhury2• 6 years ago

They don’t need google scale, they just need profitability.

Oh? They did take money from Greylock and Sequoia. Don't those guys aim as high as possible?

qqii• 6 years ago

Rather than try to build a search infrastructure from scratch, Neeva instead opted to use Bing's search API for its basic results.

So it's another UI over bing like duckduckgo? I'm not too optimistic, at the moment there are fundimental issues with how search engines interpret text and rank results.

RyanShook• 6 years ago

What’s the point of using Neeva if it’s just going to serve me Bing results with fewer ads?

nmstoker• 6 years ago

Presumably it's a matter of how they combine aspects of the results, focusing on being more useful, as the article explains.

A distinct focus isn't just "Bing without ads", in a similar way that Bing isn't just Google with different ads. They can have different focuses, qualities and utilities.

tech-historian2• 6 years ago

Well, you get to pay for the privilege. That seems to be the difference.

marcell2• 6 years ago

Is DuckDuckGo just a Bing UI? I thought they had their own index.

quotemstr• 6 years ago

Only for their infoboxes. The organic results are just Bing.

jameshush• 6 years ago

DDG USED to be Yahoo (which... Is just Bing results too. Yup I’m serious). There’s a bunch of alternative search engines that are UI’s on top of Yahoo/Bing. Ecosia is one of them too.

mhaberl• 6 years ago

DDG is not a UI over Bing

https://help.duckduckgo.com/results/sources/?redir=1

skrebbel• 6 years ago

It mostly is, though.

kinkrtyavimoodh• 6 years ago

Unless they are maintaining their whole global index, it IS a Ui over Bing. Don't be fooled by their 400+ sources, that stuff just affects things like the answer box

mda2• 6 years ago

It is mostly.

Kiro2• 6 years ago

We also of course have more traditional links in the search results, which we also source from multiple partners, though most commonly from Bing (and none from Google).

Sounds like a UI over Bing to me.

yalogin2• 6 years ago

If every other "search engine" is using Bing, is it profitable?

rcardo11• 6 years ago

Search engines are such a big thing they should be open sourced and distributed over the community. It's like the most basic infrastructure the internet needs to work and we are outsourcing it.

jeffbee2• 6 years ago

Please describe in detail how you will distribute crawl, index, and ranking as basic infrastructure.

rosstex• 6 years ago

Sounds like a lovely PhD dissertation!

WrtCdEvrydy2• 6 years ago

The same way you decentralize anything else.

You can do crawling by using an extension that allows you to create a new tab, crawl data on your current url and send it up to the mothership.

You can actually do even better because you don't get SEO-hacks like disabling certain javascript when Google is on the page to improve speed.

searchableguy• 6 years ago

https://commoncrawl.org/

Obviously, not at the same level as google and there are other parts. But I believe we can do this together if we try to. People were talking about building their own search engine on elixir forums a while ago and many seemed interested.

pabs3• 6 years ago

That sounds a bit like yacy or searx:

https://yacy.net/ https://searx.me/

c3534l• 6 years ago

If you could return search results that actually contain the keywords I entered and not waste 5 minutes of my time returning irrelevant results because you only found 3 relevant results (you know, the ones I was actually looking for) to pad out the results, that would be a massive improvement. Search has gotten so bad that I feel like the only reason no one has toppled Google at it is because search engines are almost passe. No one cares about them much any longer because people go to a handful of site for specific types of content rather than find data scattered and unorganized across the web.

non-entity• 6 years ago

This! So much. I'd rather have a search be truthful and return no results than return shit I dont care about. This applies to many things outside google as well.

Ozzie_osman• 6 years ago

I hope this works out. I'm a big believer that companies eventually take the shape of their business models, and if you're free but serve ads, you're an ads company, not a search company. So it'll be interesting to see how a company without the ads tension ends up evolving.

That said, search is a really hard problem to solve (even if you can take shortcuts like using Bing's API).

protomyth• 6 years ago

Using Bing's API isn't really going to help with some of the problems.

One thing that would be really helpful, stop counting the words appearing on links on the sidebar or other non-content part of the page as important for my search. It's amazing how many searches go astray because someone has some words on a sidebar that don't have anything to do with the content of the page. You would think with all this ML someone would teach a search algorithm to ignore it.

Twisell• 6 years ago

Was filled with hope at the start of the article and it faded away pretty quickly while reaching the part about Bing. Thereafter hope suddenly fell from a cliff...

Neeva's most unusual feature is its ability to also search users' personal files. In a demo, Ramaswamy searched for tax documents and photos, all surfaced within his search results or available in a Personal tab in the Neeva interface.

Privacy focused my a$$!

kasabali• 6 years ago

That, and forum post signatures.

Search results for hardware problems üs a total garbage landing you in forum topics of totally different hardware, just because some show off poster listed every single electronic equipment he's ever owned in his forum signature.

sushshshsh• 6 years ago

How would somebody write their own search engine?

I'm imagining some access patterns like this:

If I know what site I want to get an answer from (youtube), i can just download lots of text from youtube and search those text files for the string im looking for, making a map of files that match that string (or youtube could expose an API for me to use of course)

if i dont know what site i want an answer from, well this becomes harder, and presumably if i dont have the space to store a text only copy of the internet for me to grep through, then a 3rd party web copy (such as wikipedia) is probably a good starting point???

titzer2• 6 years ago

Hey, this is great. Sridhar was my VP for 3 years of my tenure at Google. Hell of a nice man. Great that he got out of the ad business and I absolutely think this a great thing to be working on!

person_of_color2• 6 years ago

Are you still working on ads?

titzer2• 6 years ago

No, I worked 6.5 years more, on V8 and WebAssembly and I quit Google last August.

lettergram• 6 years ago

Having actually built a different kind of search engine (see https://hnprofile.com)

I’m kinda miffed that all these posts use bing. I’m sorry, that’s not a new search engine, it’s a new UI.

moralestapia• 6 years ago

How to build a good search engine:

1. Actually return the results matching the words that people typed in your search box. The more they match, the more they go up.

More and more, it has become extremely disappointing what you get back from Google (and others as well). Verbatim search (where you surround exact terms in quotes), seems to have vanished in the last year or so. More often than not I have clicked on a result, only to find out that my particular query is nowhere to be found on the site, but some "related stuff" does.

sixQuarks3• 6 years ago

I don't rely on Google anymore when I'm trying to find good information. I actually have much better results searching Reddit and HN for specific information.

electrograv2• 6 years ago

I was not aware that HN had a search feature. Link?

wutbrodo4• 6 years ago

I share your frustration, but the obvious rejoinder is that most people don't agree with you. You can certainly target a relatively niche group of power-searchers, but most Google users probably think that the search engine guessing their intent is part of what makes it "good".

HaloZero• 6 years ago

Ha. My parents actually do google.com just to type in something like facebook.com which searches and they find facebook.

I doubt getting them to switch would be very easy unless you did some deal with browsers. But they like most others else uses Chrome, doubt Google would do that.

WarOnPrivacy6• 6 years ago

most people don't agree with you. You can certainly target a relatively niche group of power-searchers, but most Google users probably think that the search engine guessing their intent is part of what makes it "good".

Translate: Feedback from catering to the least common denominator boosts techie self-esteem.

PSA: Similar reasoning is responsible for political ads.

xeeeeeeeeeeenu• 6 years ago

1. Actually return the results matching the words that people typed in your search box. The more they match, the more they go up.

As virtually all early search engine developers have found out the hard way, it's extremely easy for website owners to cheat such a basic algorithm by spamming keywords.

Anon4Now• 6 years ago

But the answer to that shouldn't be to return results that don't even contain the words searched for.

tech-historian2• 6 years ago

Your search engine is a spammer's dream. There's a million other things to consider. No thanks.

tobyhinloopen• 6 years ago

Pagerank solved that, right?

trissylegs• 6 years ago

Just recently I was googling nftables (A replacment for iptables in Linux). Google decided iptables was a synonym and all the results were for iptables. (As it's older and more common)

azinman2• 6 years ago

Try “nftables” in quotes. It’ll help prevent query expansion.

extempore• 6 years ago

FYI there’s a search mode called verbatim, and it’s not the same as putting the individual words in quotes. It’s hidden under the search tools menu. In my experience the results it provides always contain the search terms.

kwhitefoot3• 6 years ago

Why on earth is that not the default?

bambax• 6 years ago

Actually return the results matching the words that people typed in your search box

That wouldn't take care of SEO spam, which is very good at stuffing its pages with whatever words people search for.

Also, as much as I hate to admit it, Google is pretty good at guessing wrong spelling or synonyms and getting good results in a majority of cases.

echelon• 6 years ago

How hard would it be to assign websites an actual reputation again?

Include ads on the destination website as part of the reputation score. If a website is loaded with ads, sink it to the bottom. Google doesn't do this because they likely make money from the ads.

Heuristically determine spammy content. It's pretty easy to tell at first glance which content is bullshit, so it's probably not hard to create an ML model to do the same classification.

Manually assign positive weights to websites used by engineers and domain experts. You could even curate this list in the open and solicit help in maintaining it.

amelius2• 6 years ago

The main problem: SEO.

sixQuarks3• 6 years ago

These paid services never seem to get off the ground. It's asking too much for someone to pay for something right off the bat.

Not saying paid services will never work, but I feel something like this could only work if they make it free from the beginning, actually offer better search, build a rabid fanbase, then start charging for it.

harikb• 6 years ago

I would think it is better to start with a smaller amount - not zero just to make sure you are getting users who would be willing to pay the real (higher fee) later. Subsidizing to zero only works for those which either have a network effect or ad-supported.

sixQuarks3• 6 years ago

Studies show its at least 20 times harder to get a paid conversion than free conversion regardless of price (example being free app vs 99 cent app downloads)

I think it will be easier to convert more than 1 out of 20 from free to paid once they see the value of the search

ridaj• 6 years ago

These two guys are super smart. But, how are they planning to make money?!

okaleniuk• 6 years ago

I'll pay them some.

cutler• 6 years ago

Asking users to pay for something they've been getting free for 22 years is a poor business model. Look at news websites.

puranjay• 6 years ago

But people are still willing to pay for news when it's done right - see NYTimes as an example

freediver2• 6 years ago

I have to admit this has a great title.

I am familiar with Neeva and listened to the podcast with its founder which I recommend to people interested in the field.

https://podcasts.apple.com/us/podcast/sridhar-ramaswamy-your...

I am a bit skeptical about the whole thing for two main reasons:

- Sridhar spent 15 years building the behemoth that is Google's ad business. It has to be hard to shrug off 'don't be evil until you do' culture and mindset that easily. - I think they can do a good job at specialized searches (like the one illustrated in the article). But the problem is that the web search is general, not specialized. They can not expect the users to pay $30/mo and still go to Google for general searches. Bing API results are subpar to Google results.

I do welcome a paid model for a well executed search engine. I would personally pay $100/mo for one that meets my needs.

niftylettuce• 6 years ago

Yet another startup with product that doesn't fit the market.

WarOnPrivacy6• 6 years ago

Want to build a superior search engine?

OBEY YOUR FREAKING BOOLEAN OPERATORS (search syntax, whatever) - something Google, DDG, Bing, et al, no longer have any interest in doing.

vogre• 6 years ago

That will bring you like 17 loyal users.

WarOnPrivacy6• 6 years ago

Why would you assume only 16 other people have ever had to search for something precise?

I think that number is a lot closer to everyone.

toohotatopic• 6 years ago

Google can beat them by also offering a paid-for search engine. People trust Google with their mail and phones, they will also trust google with privacy-respecting search.

On the other hand, how come Cliqz has been shut down instead of being sold?[1] Are there no companies with deep pockets who are interested in containing Google's revenue besides MS? E.g. since Cliqz was so privacy focussed, wouldn't that have been a great start for Apple to have a privacy respecting search engine?

[1] https://news.ycombinator.com/item?id=23909484

sterlind• 6 years ago

I think Apple Maps probably left a sour taste in their mouth about running such a service. Search engines are never done; it's a wide-open problem domain with diminishing returns and investment in many directions. I think Apple also realizes they're not a services company.. they sell hardware and support an OS and app store; everything else is a value add to bring people into the fold.

the_other• 6 years ago

Apple is absolutely building core business around services, and has been for several years.

Siri has search and its top hits are surprisingly helpful. Maps is getting slowly better, and has both native and web clients. Apple TV+, iCloud Drive has paid tiers. Shortcuts and Messages look like apps, but they’re really UI around services, as is Siri.

Doctor_Fegg• 6 years ago

Apple does have a search engine. Lots of the search suggestions that appear when you type in the Safari omnibar are Apple-sourced.

marmshallow• 6 years ago

Lately I find myself inserting "site:reddit.com" at the end of my Google searches. Most of the stuff that shows up by default is pure garbage that is riddled with ads and doesn't answer my question quickly.

kannan_ak• 6 years ago

Same here. From basic questions to daily struggles, I go to reddit. Maybe I can relate more to the opinions of common people than the so called expert advice in the websites.

redisman• 6 years ago

It’s all blog spam these days. Feels like reading AI generated junk that tells me nothing of value

izacus• 6 years ago

Funny enough, people on specialised reddits can be much better experts than random blog spammers.

pvida• 6 years ago

Yes, same here. I noticed reddit is much better at providing (useful and direct) answers.

I think part of the reason for that is that off-reddit there's so much emphasis on SEO and analytics (trying to hit all the right keywords, linking to other pages/sites, tricking people to stay on a page/site longer etc.) that everything becomes very cluttered very quickly.

prox• 6 years ago

I switched to DDG mostly, and nowadays its results are great for most cases. Imo Google is a habit to break.

LoSboccacc• 6 years ago

I switched to DDG a while back when I jumped out of chrome when was threatening adblockers, but I'm not always satisfied with the search results:

it's great if I know what I want to search (i.e. I know the field and the keywords and I need the specifics) but when I don't know what I want to search I can't use it for discovery (i.e. when learning new things and I lack the terminology)

matham• 6 years ago

I have been using DDG a little more frequently recently, however, it's image search was useless whenever I tried it.

burgerrito• 6 years ago

The problem with DDG is that it doesn't do really well for non-English results. I use DDG now, but sometimes I do have to use !g to redirect to Google.

BMSmnqXAE4yfe1• 6 years ago

Used DDG for two years. Now switched to Qwant. Much better and not US based.

spideymans2• 6 years ago

Anecdote 1: I was Googling a lot of information on Angular, back when I was first introduced to the framework a year or two ago. I'd say that well over half of the content surfaced by Google was SEO spam: marketers masquerading as tutorials, with the sole intent of upselling me on some shitty Angular plug-in. A lot of them are clever about it too; they don't upsell you until you've invested a decent amount of time reading their "tutorials". The SEO spam only went away as I began entering more granular search queries, as my familiarity with the framework improved.

Anecdote 2: I have a close friend that is a high school teacher. She's not all that tech savvy, so I help her out sometimes. Google is damn near useless when it comes to helping her develop her courses. Not exaggerating at all... maybe 60, 70... 80 percent of the results for educational queries are SEO spam. Every result is essentially, "want to learn how to write a monologue? Click here to pay $20/month for the privilege".

It's gotten so bad that I jokingly tell her that it would be more efficient to just walk down to the library and take out a book on whatever topic she's querying. But, you know, they do say that every joke has an element of truth to it...

globuous• 6 years ago

A friend of mine not too tech savy always clicks on these sponsored google links that take a page and a half on some screens when looking something up. I always tell him "you clicked on an ad !" and still, he always clicks on the 2-3 ads showing up before the results, even years after. Mind not that he's a UI/UX designer, his job is attention to detail, yet, he still cannot discriminate between results and sponsored links on the google search page. I wonder how many people do that, but it must make Google a lot of cash !

scollet• 6 years ago

Similarly: try comparing big data warehouses/tools by use case.

II2II2• 6 years ago

Searching for teaching materials or activities for children is so bad that developing the materials yourself is usually far less work!

I doubt the declining quality of search results is a product of Google's own advertising business, as the article would have us believe, and is mostly a product of third-party SEO spam. This outcome also wholly predictable. Part of the appeal of Google to early adopters was the lack of the SEO spam that destroyed the utility of earlier search engines. I recall conversations at the time when Google was so new they eschewed graphical ads that foresaw this outcome, though our predictions may have been a bit on the pessimistic side (i.e. Google held up against SEO longer than we thought it would).

nuccy• 6 years ago

It seems personalization/bubbling is assisting this garbage searches. From my limited observations of close friends and family, different people tend to get different number of useless results for the same query string. I personally disabled all possible personalisations in all google products, including history of searches, youtube views, etc and it seems to actually help, at least I really don't find myself complaining on google quality at all recently.

thu21112• 6 years ago

Are you defining all paid content as spam? Or all sites that sell products?

yencabulator• 6 years ago

I know some people who sell their teaching material on https://www.teacherspayteachers.com/ -- that might be worth a shot. Sure, you're still paying, but it's honest & upfront, the prices seem very reasonable to me and on the average the material is better than random searches would find.

asddubs3• 6 years ago

I've been doing this too sometimes. i swear google has gotten worse. sometimes it just ignores half the search terms, even if you put them in quotes. and yeah, I also get a lot of SEO spam results

stabbles• 6 years ago

How to google: make duckduckgo your default search engine, enter "[term] !g site:reddit.com".

Hates_• 6 years ago

DDG has a Reddit bang which does the same: !greddit

voqv• 6 years ago

Google used to have a forum tab for search results with only user generated content among discussions and forums! Damn, I still remember how sad and frustrated I was on the day they removed it with no explanation.

blisseyGo2• 6 years ago

I find google's search filters are 100% broken. Just tested this myself again.

Try searching for

site:reddit.com best headphones

And click "Tools" and set the date filter to be "Past week" for example.

Click the first result. You will notice it's from a year ago. Click the second result, it's from 2 years ago and so on. Not a single result matches the date filter. Similar issues with the other filters.

It's 100% broken and been this way for at least a year now as far as I remember.

joking• 6 years ago

The google filter means modified on the last 2 weeeks, and dynamic pages have related posts and other part of the pages that change, so those filters are not going to work well on something like Reddit.

dpacmittal• 6 years ago

I tried it just yesterday when I wanted some real reviews for a product and it worked well.

XCSme2• 6 years ago

I also noticed the same when it comes to Reddit. I usually search for a specific problem, for example "sony TV sound issue" and want only recent results (TVs that use the latest firmware). The Reddit results appear as "1 day ago" even if they are 4 years old.

coronadisaster2• 6 years ago

Google search started to decline in quality since the release of GooglePlus (since they removed the + operator and replaced it with double quotes).

eitland9• 6 years ago

Double quotes existed before as well. + and " had different meanings but were combined into one.

And yes, it was around that time that search quality took a nose dive for me.

ma2rten2• 6 years ago

Maybe if enough people do the same, reddit will start attracting the same kind of SEO spam. It's almost like Google is the victim of its own success. If everyone try to game your system some will success.

rwmj• 6 years ago

I was just thinking a few days ago: Google is starting to resemble the last days of Altavista, where the results were full of junk.

If you use Firefox, some results that are full of a Firefox-specific scam where "you are the billionth search result" (maybe it also happens for other browsers). This has been going on for years now - I noticed it because scammers were scraping my blog for content and republishing it but making searches terms redirect to scam sites.

DominikD• 6 years ago

I'm not sure I'd search on reddit if I wanted quality results. ;)

ur-whale4• 6 years ago

Actually, Google has gotten so bad that stuff on reddit is more informative. That's got to tell you where Google is at this point.

chris_f3• 6 years ago

I made a web search tool [0] that pulls Reddit results (and other sources) directly into the SERP.

I'm working off the thesis that combining highly relevant vertical results is the best way to combat SEO.

As a fun aside, when I did a Show HN for this tool, the title was "Runnaroo, a new search engine that didn't raise 37.5M to launch" as a friendly joke toward Neeva. Dang changed it later to be just "Runnaroo, a new search engine."

[0] Example search for "bose headphones reddit": https://www.runnaroo.com/search?term=bose%20headphone%20redd...

tifadg1• 6 years ago

sort by date doesn't work. Neither does on google, but if you're trying to get me to bookmark, what's your distinct advantage over the easy to remember site:reddit.com?

volkk• 6 years ago

i've found runnaroo to be super helpful with programming related results where i can actually find useful blog posts instead of those spam SEO low quality ones. awesome site. thanks

stOneskull• 6 years ago

i love this. in the article, it's mentioned how one idea is stackoverflow results, and then i see that that's nicely done in runnaroo. as well as others depending on the search. excellent search tool. cheers!

43920• 6 years ago

This is an interesting idea! It's something I've been thinking a lot about recently, so it's nice to see someone actually do it.

I saw from your previous post that you're using Google for the web results, but the only option listed in their docs has a 10k queries/day limit. Have you been able to get them to agree to a higher limit, or are you planning to move off of Google once your traffic grows?

Also, your example search has a character encoding issue - "Stolen iPad Pro & Bose headphones".

IAmGraydon2• 6 years ago

I’ve been doing the same for a while, but Google also seems to be actively limiting and censoring reddit results as well.

jerome-jh• 6 years ago

My main grief with Google is that it frequently omits keywords, contrary to DDG. As to trying to make people confuse organic results and ads, Bing is by far the worst. It often proposes malware sites when queried with open-source projects (try VLC, Audacity).

Sure having a search engine do exactly what the user requests would be great for us, IT people. Not sure it is possible to build much traffic with that alone, though.

bsdubernerd2• 6 years ago

DDG also omits search terms, sadly. It just does so a bit less blatantly than Google.

arbirk• 6 years ago

This. Google is in a bad state after their switch to the nlp engine. Now when you search for very specific things you end up with list of all phone number combinations or some other spam

nklayer• 6 years ago

And if your queries are too specific while not being logged in, you get the "unusual traffic" captcha and can volunteer to improve Google's image recognition programs.

SquishyPanda23• 6 years ago

!r on DuckDuckGo is slightly less typing and easier on the muscle memory.

I use !g for Google pretty frequently, but there are many searches I just send directly to the site I think has the best answer.

user00012-ab• 6 years ago

which is weird, because I have never found a useful response on reddit. I find people asking the exact question I have, but then there are either no responses, joke responses, or people also asking for help.

I'm still tricked every time though, I see the result matching exactly what i'm looking for, I disable my reddit block from browser, click the link, and am 99.9% of time disappointed in the result.

m-i-l• 6 years ago

Not mentioned in the article or in the comments so far, but Neeva has raised $37.5m in funding[0]. I'm curious how that money will be spent, if they're not actually spending it on building a new search engine. Is it going to be mostly spent on buying the results in from Bing, and/or on advertising their new ad-free search, and/or something else?

[0] https://tech.economictimes.indiatimes.com/news/startups/7649...

ganafagol• 6 years ago

See, Google's business model won't allow it to compete with Neeva. It can't get rid of ads because ads are its whole business.

That may sound great, but it's just not true. Everybody working at a company that pays for gsuite knows this. Gmail in gsuite is not part of the ad-financed services. Instead, you pay. Google could any day offer the same thing to consumers with search. Pay some fee every month, opt out of ads. Facebook could do the same thing, btw. Boom, there goes the "novelty" of Neeva's business model.

Now comes the question why don't Google, FB etc have such offerings yet. Many people would choose them. My guess is that they would need to have quite steep pricing, if they would want to 100% compensate for lack of ad income for each user who pays instead. Nobody would want to pay FB 50 USD every month just to see their uncle's Trump propaganda. That would be quite the eye opener about how much each user is really worth to FB or Google in money terms.

Maybe services like Neeva create a market though and push FB and Google to at least consider it.

dragonwriter• 6 years ago

Now comes the question why don't Google, FB etc have suh offerings yet.

They don't have them because the people that would pay anything at all for them are largely also the people that are most valuable to advertisers, and they wouldn't pay enough.

SomewhatLikely• 6 years ago

It's a very intuitive explanation, yet the costs to remove ads from Hulu or YouTube are quite reasonable. It's possible search is a different beast I suppose. I know a lot of the search ads I click on are actually defensively purchased. The number one organic result bought the top ad slot just to defend their position. That feels far more like rent seeking than the ads on the video platforms.

simonkafan• 6 years ago

Why another general purpose search engine? How about a niche, for example a search engine that can be extensively controlled by using Boolean algebra, Regex, first-order logic?

Googles results got less and less reasonable over the last years.

wongarsu• 6 years ago

Google used to understand Boolean algebra, like (hackernews OR reddit) AND search engine

throwaway_pdp09• 6 years ago

Grep through your HD. How fast? Now imagine grepping through the internetz.

As for FOLogic, IIRC RDF was related to that (or was that, can't remember) and it takes the sites investment to get that working on their site ie. you can't expect formal relationships between objects to appear magically, someone has to do it. And FOL if it existed may be computationally expensive.

And once you've got all that, how easy would it be to use? Probably most would go back to keyword searches.

sawaruna• 6 years ago

Maybe the pendulum could swing back the other way slightly and we could have room for an older Yahoo-style indexed search engine. This wouldn't be something that would be browsed instead of search, but if you had a general categories list indexed, say 'finance' or 'graphic design', users could enter those sections and search sites categorised accordingly. Perhaps some ranking could be done based on search terms used within a category and what site a user ends up visiting (e.g. many users end up visiting a certain stackexchange link when searching for 'parsing json in python' within the 'computer programming' category, and so its rank increases). Heck instead of ads, maybe each category page could have a list of 'popular / trending sites on this topic' section at the topic that pages could pay to be placed in.

Not sure if this solves any user problem to be honest, and the idea is only appealing to me because I think some amount of domain expertise and human curation could go into categorising pages. While this sounds (and no doubt would be) labour intensive, if we consider the number of domains that users actually visit when conducting a search (i.e. ignoring anything past page 1 of google) then perhaps it's not so extreme.

eric4smith• 6 years ago

Back in the good old days of Dmoz, this was the way, but then it became extremely limiting, and I remember wanting to rank in the directory for something and I "knew" one of the people in charge of a section. There was even some accusations of "pay for play" too. Eventually Dmoz died when Google came on the scene. But I get what you mean.

nine_k2• 6 years ago

So, finally a paid search engine that does not have to rely on ads for revenue.

So far it's based on Bing, which does. This makes it a bit a hard sell, compared to an intelligent ad blocker.

The most important problem of search engines is SEO spam. Google themselves sort of have a moral hazard to not be too stringent on SEO spam, because it shows ads by Google, increasing Google's revenue.

OTOH I wonder if the subscription revenue is going to be sufficient to have access to a reasonably good search index and enough processing power to efficiently combat SEO spam while returning relevant results. This takes your own data centers run frugally, because fees of something like AWS or Azure will just be exorbitant for a global search engine and a global search index.

I wonder if companies aiming to provide alternative search engines will cooperate on maintaining a common index, to distribute the massive costs of doing that. They could even publicly sell access to it, at a point where running a competing search engine won't be practical; e.g. researchers would buy it.

ma2rten2• 6 years ago

I am actually wondering if search engines really need such a large index. The vast majority of sites in Google's index are crap. If you were able to better select which sites to index it might help search quality and improve efficiency.

nine_k2• 6 years ago

How can you tell what is crap while crawling, is there enough time? How can you tell what is crap for the next person, and what is not crap?

It's not an easy question.

BitwiseFool• 6 years ago

This is just a hunch but I imange the internet also follows the pareto principle and at least 80% of everything in google's index is basically worthless.

We may end up with search engines that use multiple indices where each is curated to a certain domain of information, rather than a one-size-fits-all index.

a3camero• 6 years ago

I'm working on a search engine like this. So far just government websites but I think you're right about the need to focus on selected sites for some kinds of searches. For example, if you're looking for securities laws, you probably want to search government law sites and securities regulator sites. Maybe you want law blogs and law journal articles too, but you probably don't want Reddit or Hacker News.

bananaface4• 6 years ago

I think Google bases a lot of its ranking on bounce rate, i.e. "how often people who searched this were happy with this result." I don't think you need that many hits to establish bounce rate for quite specific searches. Like, is 40 is enough? I can imagine so.

But if you filter for paid members, you've filtered for much smarter people (on average) picking the best results - and you've also filtered out a lot of the SEO people who are going to be trying to manipulate the bounce rate with proxies.

narrator• 6 years ago

There's already a paid, curated search engine Google can't beat in certain verticals: Westlaw and LexisNexus, but they are based on vast curated and annotated data sets of legal case law that deliver enormous value to end users of those services.

scoutt• 6 years ago

The more I think about it, the more I think the search engine problem has no (currently known) solution. The options are:

1) Word-based crawling/indexing: quickly abused by spammers.

2) IA/ML-based: I think this is the current model (?), but after a while the machine got "clever" and it makes Google to think it knows better than me about what I am looking for, and returns result for "most people" tastes. The problem is when you are not "most people", and/or you are looking for some niche topic/work related/tech stuff/etc. Simply trying to discover new things like an interesting blog or a small shop it's impossible.

3) Paid-based: as in the article, and might be a good idea. But I think it has to run on a custom indexer. Why would I pay for Bing results?

4) Aggregators: a search engine that returns results from a bunch of other search engines, like DDG and others.

5) A mix of the above?

So unless new ideas come to the rescue, I think it's always going to get worse.

eitland9• 6 years ago

1) Word-based crawling/indexing: quickly abused by spammers.

I guess that ends quickly once spam means you get blacklisted no matter how many Google ads you serve ;-)

A large piece of this article hints at how they have some interesting options that Google doesn't have.

amanzi• 6 years ago

I want to share a recent search I did through Google -- at work I wanted to look up how to implement conditionals in Microsoft Forms (i.e. the next question in the form is based on the previous answer). I searched for "microsoft forms conditional" (without quotes) and the first search result on Google was this page on how to "use branching in Microsoft Forms" - https://support.microsoft.com/en-us/office/use-branching-in-...

That page doesn't contain the word "conditional" at all - the word that Microsoft uses is "branching" but Google deduced that it was the best result, which was perfect. The same search in either Bing or DDG produces results that all have the words "microsoft", "forms", and "conditional" in them and none of them link to the page I mentioned above, which I consider to be the best result.

Moral of the story? Search is hard, but Google does it better than everyone else.

Also, I learned via this thread that DDG is just a wrapper around Bing, which explains why the search results between DDG and Bing were near identical - they even have the matching video suggestions.

dunefox• 6 years ago

I never understand why people seem to like the DDG results so much. I often have to go back to Google.

sshagent• 6 years ago

I was moaning about this elsewhere on this site, how searching for exact phrases in quotes seems to now be a dead thing. Google and Amazon both just pick a couple of the words and shows you that. Dammit, if i put multiple words in a quote search for that phrase. If you don't find much, then sure show the other thing.

popinman322• 6 years ago

Just a comment:

Sounds like the results of a neural network: roughly approximating your intent and searching around that intent, in continuous space, to find other viable search terms and phrases. (This is one possible approach, given in broad strokes.)

That's a massive barrier to entry. You need enough data and compute to train a massive language model, more compute to run the model against all incoming queries, and then even more compute to handle the extra search load precipitated by use of the language model.

Not to mention the years of R&D that go into these models and their associated tooling.

eitland9• 6 years ago

That's a massive barrier to entry. You need enough data and compute to train a massive language model, more compute to run the model against all incoming queries, and then even more compute to handle the extra search load precipitated by use of the language model.

Luckily, most of the time you could improve my user experience by removing that cr*p and give me my 2007-2009 Google back.

From there you would only need to allow users to make personal blacklists, share personal blacklists (this was about the time when auto-generated content started to become popular) and maybe also aggregate some popular blacklists for a default blacklist and it would be better than anything we have seen since.

(I remember having a txt-file with -spammydomain.com -anotherspammer.com etc etc that I pasted in at the end of certain searches to take care of sites that had either had

- auto-generated content

- or stuffed their pages with black/black or white on white keywords )

ur-whale4• 6 years ago

Moral of the story? Search is hard, but Google does it better than everyone else.

True, but they do it worse and worse.

Google still has some margin before their usability declines to the level of their competitors, but they're headed there quite fast.

BLanen• 6 years ago

No, it's deliberately made worse the last couple years. They have put way too much confidence in their semantics-ai and it seem way overfit.

I used to be able to learn how to effectively search but lately it's just so terrible. It's made for 99% of people's dumb searched, but try to get specific and it fails hard.

habosa• 6 years ago

I just can't get past the narrative here. Sridhar basically builds Google's massive ad business from the ground up and then decides "ads are bad" and takes his ad money over to profit on the other side by selling a new search engine without ads.

Reminds me of "The Prodigal Techbro": https://conversationalist.org/2020/03/05/the-prodigal-techbr...

kwhitefoot3• 6 years ago

What I want is a search engine UI that will guarantee that my search terms appear in /all/ of the hits. And allow me to specify terms that must not appear and guarantee that the hits do not contain those.

I'm semi-seriously considering writing my own.

moritonal• 6 years ago

Not to seem rude, but do you know about "https://www.google.com/advanced_search"?

kwhitefoot3• 6 years ago

How do you find it if you aren't aware that it exists. There is no longer a link to it on the Google search page as far as I can see. And, as soon as you hit the search button you are back in the ordinary search so refining the search is tedious.

The UI stinks.

However, I am grateful for the reminder.

est• 6 years ago

Search engines are dead because public websites no longer carries useful information. Tons of user creations are on mobile now, in walled gardens.

We need hostable websites on mobile phones. Or data will be centralized to few big corporations.

shafyy• 6 years ago

From their website:

Neeva finds exactly what matters to you, whether it’s on the web, buried in your emails, or deep in an impossible-to-find document.

Why not stick to just a good search engine? I don't want to hook up my email accounts cloud storage accounts to this.

Usually people know if they need to search on the world wide web or in their own emails or docs, don't they?

tomxor• 6 years ago

it felt like a product being made worse by its business model.

Something even worse than this has has forced me to ddg out of necessity: fucking CAPTCHA.

Which is essentially discrimination against anyone with a mobile internet connection who blocks google's tracking.

I just can't use google search anymore, it's a special kind of torture, after years of getting used to using google for everything and now getting this thrown in my face every single fucking time - way to permanently train users away from your search engine. It's time we had more competitors, google search is a monopoly and it's only going to abuse it's users more.

ur-whale4• 6 years ago

A few things you need to build a search engine:

    1. something that *understands* the question
    2. something that *understands* content (web pages)
    3. something that can match the output of 1 and 2 very fast and at scale
    4. a way to generate revenue out of the whole thing without bothering the user or perverting the output of 1,2 and 3

In the case of Google, they have a fairly good implementation of #1,#2 and #3 and they used to have found a decent compromise for #4.

In the last 5 years though, they have completely eff'd up #3 because of the pressure that comes standard with large US-based corps: the next quarter EPS is the only thing that matters, and long-term growth is not something wall street is interested in.

Burn a patch of forest, grow something on the ashes-fertilized soil for a few years and move on to the next patch when the soil is permanently dead.

m3kw9• 6 years ago

Once I saw subscriptions based search, I cannot bet on it just because when I search I can filter the ads myself from google pretty easily, and it is quite hard to beat googles search accuracy.

amelius2• 6 years ago

I'd like to see a curated list of papers for building a search engine.

Aeolun• 6 years ago

They’re not building a search engine at all. They’re another duckduckgo, but charging a subscription fee.

They’ll get my money when they’re actually building a search engine.

mark_l_watson• 6 years ago

If it weren't for Duck Duck Go, I would absolutely consider paying a small monthly fee to Neeva for ad-free (and hopefully tracking-free) search service. I might anyway.

Interesting that both Neeva and DDG build on Microsoft's Bing service. I also use Bing search for small personal projects. Search APIs are useful for many automated information extraction use cases.

There are paid for Google services that I really like (GCP, Play Music, Play Books, Play Movies), but I now avoid search and gmail.

hliyan2• 6 years ago

If only the early Internet had a standard search protocol RFC alongside the others, so that each website could implement it, and there would one day be search aggregators with content quality filtering.

phenkdo• 6 years ago

I think "search" needs a paradigm shift, in fact keywords -> links paradigm needs radical change. Something akin to a GPT-3 model but with interactivity and self-customization.

jhoechtl• 6 years ago

I signed up but I fear that it is going to be shopping-centric. So results are bought instead of intelligentely mined.

yalogin2• 6 years ago

I agree with the sentiment that google search results are not what they used to be. And its google's fault. Not as much because of their search algorithms as it is their ad business. Talking pure about tech articles on the internet, Google's ads have corrupted the previously benign and even helpful blogs and other sites and turned them into ad peddlers. I mean who wouldn't want a recurring revenue stream?

That kind of reached its zenith (or nadir) when companies started sucking in articles and employing people to write tutorials just to get ad dollars. Every one of the tech articles. I am sure the other fields are the same way. So unless you know a site you want to search in, the results are all ads not real content.

bhartzer• 6 years ago

The problem is ads and ad revenue, but Google is increasingly scraping content from websites in order to put that content on their own site—To keep users on google properties as long as they can.

ralmidani• 6 years ago

I would gladly pay $5 a month for a totally ad-free and tracking-free search engine that lets me easily block sites and/or downvote results that aren’t relevant to me. I already pay for Apple hardware and Protonmail, and use FF and DDG whenever I can, in no small part to avoid Google and MS. I would love to see DDG or someone else offer a no-nonsense, “premium” search engine.

LockAndLol• 6 years ago

They could be using YaCy, setup their own infra and provide premium access to fast and more reliable YaCy results. That would at least make them a competitor and not just a Bing UI.

I mean, sure, it's cheaper, but it doesn't really set them apart from any other Bing UI. Actually running something they own, especially something opensource and federated, would distinguish them and warrant whatever fee they deem necessary.

wackget• 6 years ago

This website (protocol.com) hounded me with a newsletter subscribe popup as soon as I landed on the page.

I hadn't even had a chance to read the first line of the article and the site's already asking me to sign up for their marketing trash.

Why in the world would anyone think that's acceptable? I have never seen the site before and have no idea of the quality of its content so what on earth makes them think they're going to get a subscriber out of me?

XCSme2• 6 years ago

I agree, I usually insta-close a website when I get an unexpected pop-up. Pop-ups should only be used as a result of a direct user action (eg. click on delete account, receive a confirmation pop-up).

sid-• 6 years ago

I think we should have subscription products that respect privacy and data for all successful products; Maps, Youtube, Search, FB, etc.

135792468• 6 years ago

As an SEO, I’m embarrassed how accurate the comments in this thread are. I’ve had this conversation with with peers but the search results are utter trash and it makes me feel bad to be part of it, finding good information is so hard.

Until Algos get smart enough to understand text at a deeper level we will not get anywhere.

We really need more competition. I know Apple gets billions from google for being the default search but if they were to buy DDG which results are mostly goof enough, they’d be a viable competitor. Until something big like that happens, Google will continue to be below average

IndexPointer• 6 years ago

So, another Bing API wrapper to add to the list?

One company's plan to build a search engine Google can't beat

Related Stories