Searx – Privacy-respecting metasearch engine

https://sagrista.info/../../../blog/2021/searx-or-duckduckgo/

by cube00 • 5 years ago

299 147 5 years ago

Qwant is another (non-US) alternative

rahen2• 5 years ago

Doesn't it use Bing results, like DuckDuckGo does? I read it's hosted on MS Azure like Bing and DDG, so in the end it's somewhat just a rebranded Bing. Quite a shame for a European (Franco-German) search engine.

kaba02• 5 years ago

Even if they were just a proxy behind bing, that would be still good, weren’t it?

mg4• 5 years ago

Every time I stumble across a new search engine, I add it to my search engine comparison tool:

https://www.gnod.com/search/

Will add SearX now. It seems to provide reasonably good results.

Update: It's on (under 'More Engines').

imglorp• 5 years ago

This is a nice compilation.

It would be very interesting if it examined and compared results.

m-i-l• 5 years ago

Yes, looks good, although I thought it was going to be a federated search, i.e. you enter your search term and it performs that search on all the sites selected. The simpler way of implementing a federated search would be to show separate results boxes from each site, although that wouldn't scale well to a large number of sites, and it can get quite complicated to try to combine the results.

51stpage• 5 years ago

There is also Marginalia https://search.marginalia.nu/ which I don't see on your list.

marginalia_nu8• 5 years ago

It's currently undergoing its monthly maintenance just FYI. It's up and technically working, but with a drastically reduced index size.

sysadm1n2• 5 years ago

Tried Marginalia. So many plaintext http links which I avoid like the plague. That's my only gripe with it. Other than that, it's an awesome tool.

jwithington• 5 years ago

where's you.com lol

mg4• 5 years ago

Thanks, added.

ColinHayhurst5• 5 years ago

Nice work. If you have a twitter handle you might request to get added to these lists; either way they might be useful for you: https://twitter.com/SearchEngineMap/lists

fsflover2• 5 years ago

You should also add YaCy: https://yacy.net.

mg4• 5 years ago

It seems to be not web based?

When I click on "Try out the YaCy Demo Peer", I get "502 Bad Gateway".

Jaruzel• 5 years ago

I have this REALLY old text file of search engine URLs:

http://www.jaruzel.com/textfiles/Old%20Web%20Info/Internet%2...

Google basically killed almost off of them off :(

It would be great to see some proper competition in the search space, especially around specialist search engines.

sixothree2• 5 years ago

I would love a search engine targeted towards developers. Searching for symbols seem to be a problem with google, not to mention all of the utterly crappy results they serve up.

Minor49er• 5 years ago

You should add Fireball. It's excellent

https://fireball.de/

wenbin2• 5 years ago

You should add the podcast search engine https://www.listennotes.com/

mg4• 5 years ago

Thanks, added.

Holy Moly, are there really over 100 million podcast episodes out there?

autoexec• 5 years ago

Too bad they force you to log in to view a result or do anything but search. They also share/sell your data to 3rd parties including Google

nix23• 5 years ago

A yacy instance would be good too ;)

https://yacy.net/

danskeren• 5 years ago

Ask.Moe

nona.de

1_player• 5 years ago

Is anything supposed to happen when I enter something and press Enter? Nothing happens for me, FF on Windows, uBlock.

reayn• 5 years ago

You are supposed to type something into the search field then click the engine you want to use, it will pass on whatever you entered.

I agree that the creator could make that a little more clear somewhere on the page.

GhettoComputers3• 5 years ago

Isn’t hosting your own instance taking away every benefit of searx by revealing your IP? If you had a VPN you’d use it to mask your IP from tracking of search engines anyway, and if you used Tor for it, you’d probably move back soon since it’ll be so much worst with latency, like how many people go back to google because DDG sucks for results. I suggest just using instances you can find them at https://searx.space some are more reliable than others but none have been trouble free. There’s a lot of these instances for privacy, chrome has a privacy plugin with a white eye that uses nitter.net for Twitter, teddit for Reddit and other public instances. One instance of Reddit was even made completely in rust. ;)

https://chrome.google.com/webstore/detail/privacy-redirect/p...

To reply to the person under me, you’re always relying on a trust in something unverified and untrustworthy filters like VPNs anyway, you’re either revealing your IP using a wrapper that reveals it instantly, use a site that isn’t a search engine and might be using your data, using a VPN that is based on reputable and assumptions, usually based in another country you won’t visit or know much about aside from random reviewers, or using Tor, losing latency, reasonable speed image search, and still be possibly compromised.

kleinsch• 5 years ago

But if you’re using instances someone else is hosting, aren’t you hitting half the author’s objections?

- They may be hosted in the US

- They may be hosted on AWS

- You have no idea if the maintainer of the instance is tracking you

jarvuschris• 5 years ago

^ This right here. This article is pretty hogwash IMO

Points 1 and 3 aren't relevant if they aren't recording the data. Companies in other jurisdictions have no magic invulnerability you can trust to their data getting out (legally or illegally) if they're storing it.

Points 2 and 5 are equally true of any open source project unless you run it yourself from source. There are _plenty_ of examples of users getting phished by maliciously built/hosted open source tools

Point 4 is obviously not malicious tracking and a mistake any project could make

At the end of the day though, unless you're going to run everything yourself (which most people aren't) you have to pick who to trust -- some random person running a server somewhere, or a company with hundreds of employees recruited under the premise of working on a privacy-centric search engine who could all turn whistleblower

jostillmanns• 5 years ago

Whoogle is another alternative, that focuses on Google search results

ced• 5 years ago

From the link:

The CEO sold his previous company's data before founding DDG. His previous company (Names DB) was a surveillance capitalist service designed to coerce naive users to submit sensitive information about their friends.

Is that a fair statement? Can someone provide more context?

boomboomsubban4• 5 years ago

It was a failed social network to help you reconnect with old friends. It tried to get you to recruit your friends immediately after registering and had a typical social network license. I'd say that description is intentionally describing it in the worst possible light, but not wholly inaccurate.

ColinHayhurst5• 5 years ago

https://www.searchenginemap.com/

account-5• 5 years ago

That site is horrible on mobile, a good portion of the screen is taken up by the orange "download/view" infographic thing. Interesting though to see how connected the engines are, I would have thought DDG would be bigger with its bang option though I assume it's about what is natively included in the results.

ColinHayhurst5• 5 years ago

Yes, it is horrible on mobile. The size of all the syndicating search services are the same. An update is overdue.

Complementary twitters list are maintained here: https://twitter.com/SearchEngineMap/lists

freediver4• 5 years ago

I believe rightdao.com is missing from the list. It has independent index (and also impressive speed).

Also not sure what the criteria for inclusion is, but search.marginalia.nu and teclis.com both have their own indexes.

skerit• 5 years ago

Alright, going to try searx.be for a while then.

jqpabc1235• 5 years ago

I heard it could be a honeypot for the CIA. But feel free to prove me wrong.

unixfox2• 5 years ago

If you don't want to use a single searx instance then feel free to use a random one automatically for each search thanks to this tool which can be used locally: https://searx.neocities.org/

schleck8• 5 years ago

Metager is a non-profit, open-source search engine running fully on renewable energy. It also has a proxy for opening results anonymously

https://metager.de

hermitsings3• 5 years ago

https://metager.org/ for english users

czechdeveloper• 5 years ago

That seems quite exclusive to germany

kristianpaul• 5 years ago

Doesnt searx looks for results at duckduckgo and google for you anyway? Whats the difference from using ddg directly?

sebow• 5 years ago

I don't think it uses DDG directly.But anyways you can configure the sources for files, media, wiki,etc. Makes sense since the engine is open source, but then again it's not really a search engine itself but a metasearch one

boudin• 5 years ago

There's other sources available. It is a meta search engine, so it will always rely on other sources, but you can disable duckduck go and google backends.

luciusdomitius• 5 years ago

Isn't duckduckgo just an alternative frontend for bing with an integrated ad/tracking blocker. Or at least that's what they claim.

nicce• 5 years ago

It is. They say they add some indexing on they own, but results are all the same.

BeetleB• 5 years ago

Searx can ping multiple search engines, including those not supported by DDG. For example, searx has a dedicated file search, which includes torrents.

ahtaarra2• 5 years ago

I recently made the swtich from DDG from Searx simply bevause right-clicking on a search result to copy the url resulted in a referrer link to be copied rather than the link of the result destination.

caskstrength• 5 years ago

I use https://addons.mozilla.org/en-US/firefox/addon/clearurls/ to automatically convert referrals to actual links on web pages.

ravenstine• 5 years ago

I've been using ClearURLs for a while so I never even realized that DDG used referrer links. Have they always done this?

southerntofu• 5 years ago

I think enabling DoNotTrack header or disabling JavaScript prevents this behavior: i cannot reproduce on Tor Browser. But you are correct this is a worrying development.

boomboomsubban4• 5 years ago

Isn't the referral link only on the ad results, which is clearly marked "AD"? That's what my quick test shows.

ahtaarra2• 5 years ago

Now I am not sure what exactly the issue is but this seems to be applicable to all results, at least on my particular setup.

On LibreWolf browser,

On mouse hover, the correct result urls show but then I right-click one, it shows something like:

https://duckduckgo.com/l/?uddg=<destination_url_here>&notrut...

But this same behaviour cannot be observed on Google/Searx on the same browser. It also even isn't observable with DDG on a FF nightly build on the same system and a FF stable build on a separate one.

Edit: url formatting

EDIT2: this behaviour seems to be limited to my particular setup and even a clean LibreWolf profile seems not to suffer from this issue. I apologise for the misunderstanding.

drcongo3• 5 years ago

As someone who uses Safari with its built in list of search providers, I'm rather stuck with DDG for address bar searches, but boy it has really started to suck over the last year or two.

boomboomsubban4• 5 years ago

You could always do !searx

drcongo3• 5 years ago

True! I'm experimenting with Ecosia right now, but also recently got on the beta for the kagi.com search engine which so far has proven to be vasty superior to any other that I've tried.

ColinHayhurst5• 5 years ago

Yes, Safari is the worst offender when it comes to offering search choice on desktop.

On iOS using a new app called Hyperweb you can the new Safari extensions to access and create a longer preferences list. https://hyperweb.app/

We really shouldn't have to choose this or that, but should be able to easily use multiple choices in search. You can do that today as explained here, but you'll need to switch browsers. https://blog.mojeek.com/2021/09/multiple-choice-in-search.ht...

jeroenhd• 5 years ago

with its built in list of search providers

TIL. That's just... terrible UX.

bombadilo• 5 years ago

So don't use Safari then? Seems like a pretty simple solution.

sixothree2• 5 years ago

Edge is available for Mac.

drcongo3• 5 years ago

Chrome destroys the battery on my laptop and is basically spyware these days, the Chrome-alikes are all dreadful - Vivaldi has the jankiest UI ever, Brave is unbearably sluggish, Edge runs processes that I didn't ask for like Microsoft Updater that bugs me constantly and spams the new tab screen with all sorts of low rent junk that I can't remove.

Firefox is my developing browser and I do really like it, but Safari my actual browsing browser because it's by far the best browsing browser on Macs.

ricardo81• 5 years ago

'Uses Amazon Web Services (AWS) as a cloud provider and Cloudfare CDS.'

IIRC DDG uses Microsoft servers now exclusively. Makes sense given the volume of queries they're handling and all dependent on Bing API.

hermitsings3• 5 years ago

I had been considering using Searx (which I had known about before) lately since I have to use DDG+Google for getting satisfactory results.

Edit: I really like DDG bang and vim like nav keys tho

visiblink• 5 years ago

Someone here once posted a link to duckduckstart.com. I have it set up in my search bar now.

If you search, it goes through Startpage (Google results, more privately). If you search with a bang, it goes through Duckduckgo. It's probably close to what you're looking for.

hermitsings3• 5 years ago

saw it just now. wow thanks

phantom_oracle2• 5 years ago

I will add add a disclaimer to this comment that it is tinfoil-hat and just speculation(bordering on conspiracy) but many of these "we are a privacy-first company" might actually just be honeypots and fronts for 3-letter agencies.

The comment is not wholly conspiratorial, considering the CIA owned Swiss crypto company: Crypto AG [1]

It's within the realm of possibility that most of these privacy services could be owned by 3-letter agencies or small enough to be coerced into cooperation.

[1] https://www.scmp.com/news/world/europe/article/3050193/crypt...

jqpabc1235• 5 years ago

Haven't you heard? The CIA has gone open source. They don't need to own a company anymore.

They can just download the Searx source code; modify it as they see fit, and make it available on a server someplace.

Can you prove that searx.be isn't run by a "3 letter agency"? Can you prove that the source code running at searx.be is the same as on Github?

The point being --- unless you have full access to the server, open source means nothing with regard to privacy and security of any service. It actually means less than nothing --- it means it is super easy to build into a honeypot.

marc_abonce2• 5 years ago

Of course, there's no fool-proof solution to knowing what code is running in the server side, but https://searx.space at least shows if an instance modified their client-side code, which you can see in the HTMl column.

To mitigate server-side code from identifying you, you can consume an instance from Tor. Of course, you could try to do that with any other search engine, but most of the other search engines either block exit nodes or provide incomplete functionality if you disable JS.

It's not perfect, but it may be good enough depending on your threat model.

sleepysysadmin2• 5 years ago

The thing is... lets say the CIA/NSA are tapping searx wholely or just instances. What exactly are the ramifications? I feel like they are going to be largely missing the target. A bunch of techsavvy nerds trying different search engines aren't going to be terrorists.

And even if they are? As a Canadian or someone who isnt in the USA. What exactly is the point? Wouldn't this effectively be the safest host? CIA/NSA wont be selling your private infos. They wont be sending me to a blacksite because i look at python documentation and youtube chill music.

kwhitefoot• 5 years ago

CIA/NSA wont be selling your private infos.

Why not? They used to sell cocaine after all and your info is probably rather less risky.

marginalia_nu8• 5 years ago

I do think it's a bit of a red flag.

Sort of like how most anti-tracking browser extensions eventually turn out to actually be tracking extensions. Or like how used car dealers that have a name like "honest bob's cheap luxury cars" often turn out to neither be honest, cheap nor luxurious.

GhettoComputers3• 5 years ago

Isn’t that confirmation bias? uMatrix and uBlock are reliable, the opposite being PrivacyBadger. The EFF has lost my trust before but I never assume maliciousness before incompetence. https://old.reddit.com/r/privacytoolsIO/comments/l2dges/why_...

ColinHayhurst5• 5 years ago

SearX is a project which we respect and a positive contribution to improving search choice. Consideration of how it might be being used is wise.

It's also wise to do due diligence on any company/service where you are revealing sensitive personal information. Traffic coming from Google in 2006, for sensitive medical search queries was a catalysts for us going public in 2006 on our strict no-tracking policy and we maintained that position.

We have yet to be contacted by authorities, but you'll have to trust us on that one for now. Since we don't log any personal or identifying data at all, we would have nothing to share [0]. You can read about our investors on our blog.

Building and maintaining a search engine with independent infrastructure has a huge challenge and has meant building proprietary IP over many years. Since we refuse to use techniques used in growth hacking such as analytics from you know who, and all tools involving any tracking, marketing is a bigger challenge than it is for companies without strong principles. It has been a mammoth effort, by mostly our founder whose story you can read here [1].

[0] https://www.mojeek.com/about/privacy/ [1] https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...

phantom_oracle2• 5 years ago

I should have added that my comment implied as much to do with DDG as it does with cheap-VPN-provider-35 with a shell company in Belize.

The original comment was in reference to DDG proudly making claims of not getting requests from .gov and marketing themselves as a company who "cannot see what you search for".

no_time2• 5 years ago

Same goes for VPN companies. I do feel bad for all the journalists and whistleblowers who will fall for these scams but as far as I'm concerned, as long as I can avoid for profit data collection companies like Google, it's good enough for me.

If I can't avoid my data being collected, I will still try my best to make it as worthless as possible just out of spite.

yosito• 5 years ago

While I don't like the idea of a three letter agency honeypot, I'd be even more concerned with ad-tech and surveillance capitalism companies setting up honeypots.

ojosilva• 5 years ago

Are there any opensource real internet search engines worth looking at? I think we should be working on disrupting search as a whole instead of depending on the Googles, Baidus and Bings of the world.

I'm fully aware of the massive crawling and storage requirements, but opensource projects that can get search right can later 1) be hosted by the powerhouses of the cloud or non-profit parties, or 2) become a fully distributed hosting and crawling effort as in p2p and blockchain.

vindarel• 5 years ago

p2p: there's the Yacy effort. https://yacy.net/ I… couldn't find a portal to try it out (I did years ago, the results need… to be discussed. It's anyways easy to install and to choose what part of the web to crawl.)

YaCy is free software for your own search engine.

maybe they rebranded and don't aspire to be a complete web search engine?

YXNjaGVyZWdlbgo3• 5 years ago

I had a searx instance running for a long time and it's great when it works but the plugins for site specific searches break all the time and if you have more than 3-4 users with a high search frequency google blacklists your IP by throttling.

kaba02• 5 years ago

May I ask about what’s your idea of preserving privacy with self-hosting a website that searches for specific terms from a presumably fixed IP, where by 1/3-1/4 chance it can be attributed to you?

YXNjaGVyZWdlbgo3• 5 years ago

The instance ran on a mobile connection not associated with any private information.

EDIT: It was located in a german street light 20km away from any of the users. Just to get the geolocation question out of the way.

It was more of an experiment than anything else there will be a talk about it and other FreiFunk (Open Mesh Network in Germany) related stuff at the next virtual CCC congress.

xwdv• 5 years ago

People don’t want privacy. They want results.

Society programs us to think privacy is our top concern. Is it?

marginalia_nu8• 5 years ago

I do think you are mostly correct. Some people really care about privacy, but for most people it isn't a huge concern.

This doesn't make it any less important, but just means that if your main selling point is "we're the search engine that cares about privacy", then odds are you're not going to get a lot of users.

lardolan• 5 years ago

I agree with most of your points. Although It may be matter of a time to grow that niche. IMO Pople who value it are willing to take sacrifice of less functional outputs.

Privacy is most effective selling point when working with sensitive information.

InfiniteRand• 5 years ago

I feel like privacy is a concern relative to the chance of my information being misused and the damage from the misuse of that information. If a search engine wants to collect random facts about me but that collection is extremely unlikely to affect me in any way I couldn’t care less

goldsteinq2• 5 years ago

The problem with self-hosted search engine is that they make you _very_ unique: you’re the only client of the ”backend“ engine with that (static and non-NATed) IP. Furthermore, you’re now one of the small group of people with ”hosting“ IPs. Using self-hosted SearX may make you easier to track, not harder.

Using SearX hosted by someone else is marginally better, but now you have to trust the owner of the server, which is probably not what you want for privacy-centered search engine.

woodruffw• 5 years ago

Could you clarify where the privacy concern is here? As I understand it, I'm sharing my IP with search engines anyways; the only difference with a self-hosted SearX instance is that I'm sharing my server's IP instead.

Is the concern that the latter's IP isn't behind a NAT, and therefore is more unique? If so, I think that's the least concerning of the identifying datapoints that a search engine has access to -- my browser metadata is far more identifying. With SearX, that information doesn't get forwarded (IIUC).

mekster2• 5 years ago

Server IP can easily be differentiated from casual internet users' IP, and since you also don't usually get a server IP rotated, you're pretty much stuck on a single IP and you're easy to be categorized as an unique user no matter which device or location you use it from.

randomsilence• 5 years ago

When you click a link on your SearX instance and you don't use referrers, how can anybody track you? Nobody knows that you are coming from your "backend" engine.

You just reveal your search queries to the hosting provider if he maliciously intercepts them.

marc_abonce2• 5 years ago

If you don't want to expose your IP address, you can configure searx to proxy all the queries through Tor. This obviously makes the instance way slower and you'll have to disable some engines that block Tor exit nodes, so it's a trade-off.

mekster2• 5 years ago

Probably better to let your server go through some public VPN service.

goldsteinq2• 5 years ago

Tor exit nodes are public and given that SearX traffic is easily distinguishable from normal traffic, you're once again in a very unique group of people who use SearX + Tor setup.

3np3• 5 years ago

What you can do is:

* Share your instance with some friends (though of course it shifts the trust one level down)

* Route outgoing requests through tor / some VPN. In a hosted environment that you configure once for "everywhere", it's more feasible to do fun things like "Google searches over OperaVPN, bing searches through Mullvad, everything else through Tor". You could even change proxies.

Especially with the latter you can kind of "eat the cake and have it too", with some added latency of course.

As a bonus, searx can also be configured to rewrite links (yt->piped, twitter->nitter, imgur->rimgu etc) and remove tracking query parameters.

no_time2• 5 years ago

Do you know hosting companies that allow crawling and scanning? Most explicitly forbid such things as it makes their ip addresses worth less.

jqpabc1235• 5 years ago

You do not need to trust third parties to keep you private and not track your every move, which is awesome.

The only way to avoid third parties to run your own server ... but this "metasearch engine" is basically just an aggregation proxy. So every search can still be tracked back to your proxy server by Google, Bing or whoever is providing the actual results.

jefc1111• 5 years ago

I have been trialling Swisscows (https://swisscows.com/) and have found it quite useful. I have not deeply researched their privacy claims, but for now I am just trying to not use Google or mainstream alternatives.

Does anyone else have experience or comments on Swisscows' search engine? Seems like an interesting company all round.

AmosLightnin• 5 years ago

Privacy is important, but I also care about the terrible quality of search results I get from nearly all the major providers these days. Couldn't an aggregator like SearX host a machine learning layer that learns what results are more likely to be valuable to me, and ranks them higher in the results? Keeping the customization layer on my own server and improving search results would seem to be a big advantage both privacy and performance wise.

freediver4• 5 years ago

I always liked the term "search engine client" better (vs 'metasearch engine'). In essence it is a product that can connect to different search indexes.

An "email client" does exactly the same thing, connects to different email servers and we do not call it "metaemail".

edit: just realized that with the current hype around metaverse, 'metasearch' will probably be more appropriate for something searching the metaverse in the future.

herodotus• 5 years ago

Could this be installed on a Raspberry Pi? I am very happy with my Raspberry Pi-hole: would not mind adding a second pi for searching.

tandav2• 5 years ago

Just searched a couple of queries like "opencv rectangle", "python regex" - and it returned nothing

unixfox2• 5 years ago

Which instance did you use?

tandav2• 5 years ago

https://searx.thegpm.org/

ChrisArchitect• 5 years ago

Please keep title same as the actual post: Searx - moving away from DuckDuckGo

It gives more context to the topic, as in it's not just a link to the search engine itself.

agluszak• 5 years ago

Great! I wish there was a possibility to blocklist certain domains (who wants to see Quora in their results...). This should be easily implementable on Searx's side. Another feature I often wish for is searching in a specific time period. It's so annoying, for example on Youtube, when I remember that a video was released in 2011, but there's simply no filter for it.

dalf3• 5 years ago

I wish there was a possibility to blocklist certain domains

You can do that in this fork: https://github.com/searxng/searxng/blob/e839910f4c4b085463f1...

citizenpaul• 5 years ago

Slightly off topic but has anyone a good solution for removing content farm seach results in this or any engine. For example some worst offenders wikihow , forbes, business i sider.

gxnxcxcx• 5 years ago

uBlacklist fully supports blocking results from Google, Bing and DDG, and partially from Startpage and Ecosia [1]. It's been available for Firefox and Chromium derivatives for a while, and there seems to be a third-party Safari port as well [2][3].

There's a bunch of language-specific blocklists on GitHub focused on StackOverflow/GitHub mirrors and wonky machine translations, but I don't know of any mature curation effort yet.

[1] https://github.com/iorate/uBlacklist#supported-search-engine...

[2] https://github.com/HoneyLuka/uBlacklist/tree/safari-port/saf...

[3] https://apps.apple.com/us/app/ublacklist-for-safari/id154791...

analyte123• 5 years ago

If you're trying to be comprehensive, a few other suggestions in rough order of their usefulness:

Gigablast.com - Has been improved recently. private.sh is supposed to be a private proxy for Gigablast, but it has been broken recently

Exalead.com - run by a French defense contractor for some reason

filepursuit.com - search for files only. Need to play around with it more.

PeteyVid.com - multi-platform video search

Wiby.me - focus on "classic" style web sites

abetusk• 5 years ago

Search engines have been coming up lately, so maybe this is a good a place as any to discuss some back of the envelope calculations.

Let's say we wanted to recreate the web index made by Google. How much cost and engineering would it take?

Estimating the size of the web from worldwidewebsize.com [0], this is estimated at around 50 billion (5010^9). The average web page size looks to be on the order of 1.5 Mb (1.510^6). The nominal cost of hard disk space is about $0.02 / Gb [2].

So, roughly, that's 75 exabytes of data (~7510^15). At a cost of $0.02 / Gb that gives roughly $1.5M just to buy the hardware to store (a significant fraction of?) the web. The Hutter prize exists [3], so maybe there's some confidence that we only need to actually store 1/10 of that, so around $150k in costs.

For perspective, that's 10 multi millionaire silicon valley types donating about $150k each, 100 "engineer types" at $15k each or 1000 to 10,000 pro-active citizens at $1.5k to $150 each (just* for the hard disk space, discounting energy, bandwidth and other operating costs).

If we try to extrapolate lowering hard disk space costs and take the price halving to be about 2.5 years with a current (pessimistic?) cost of $0.02/Gb, that's about 10-15 years before a petabyte scale hard drive is available to the consumer for $1000.

From my perspective, I would ask "why hasn't a decentralized search index been created and/or is in wide use?". My guess is that figuring out a robust enough system that's cheap enough is still out of reach. $150 might not seem like a lot, but you have to convince 10k people to devote energy just to search.

Put another way, when does the landscape change enough so that decentralized search is a viable option? My guess is that when people can store a significant fraction of the web locally for nominal cost is the determining factor. Maybe some great compression and/or AI sentiment analysis can be done to bootstrap and maybe some type of financial incentives can help solve this issue, but my bet these will only provide a light push in the right direction and the needed technology is the underlying cheap disk space.

As a side note, the worldwidewebsize.com [0] shows the number of indexed pages by Google holding pretty constant over a five year period with a sharp decline somewhere in 2020. I wonder if this is the method of estimation or if Google has changed something significant in their back end to alter their search engine and storage.

[0] https://www.worldwidewebsize.com/

[1] https://www.pingdom.com/blog/webpages-are-getting-larger-eve....

[2] https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/

[3] http://prize.hutter1.net/

gnufx• 5 years ago

For what it's worth, it's used by by the de-Googled phone /e/ Foundation at https://spot.ecloud.global and seems to work tolerably well.

m0zg• 5 years ago

Or you could use the only independent search engine that actually has its own index, and search result quality comparable to that of Google: https://search.brave.com/

itschristmas• 5 years ago

I run searx locally, have it set as my default search engine, and use it as a search engine client. It's nice to have the same interface across as a variety of search engines.

Searx is provided as a service on NixOS, which makes this all simple to run.

I do the same for Nitter, a Twitter frontend, which supports RSS and behaves well while logged out.

egberts1• 5 years ago

My search engine puts a Pi symbol in lower left corner of every web page I visited.

Searx – Privacy-respecting metasearch engine

Related Stories