YaCy, a distributed Web Search Engine, based on a peer-to-peer network

by Timothee • 2 years ago

288 97 2 years ago

DrDroop• 2 years ago

I once went to a workshop on a Sunday morning at the local makerspace to listen to someone talk about some kind of distributed search engine or something like that. One of the developers came from (I think) Germany to explain this to us the centralized sheeple. He just gave a demonstration of the thing, like here is the box you type stuff and here are the results. When I started to ask questions about how it worked an all he sort of acted annoyed saying it was all too difficult to explain. This was more than ten years ago, and yes I am still angry about it.

ssijak3• 2 years ago

At the core it was probably based on peer to peer distributed hash tables, so here you go, read the source https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...

belter• 2 years ago

160 bits ought to be enough for anybody :-)

albert180• 2 years ago

It's probably him YaCy is made by a German Dude

synctext• 2 years ago

Impressive 20 year project by one key developer.

See 20 year post in German by YaCy founder: https://community.searchlab.eu/t/yacy-vor-20-jahren/1543

ssijak3• 2 years ago

Long time ago I worked for a startup called Wowd which built distributed search engine. It was acquihired by Facebook.

On of the biggest issues was how to entice people to download and run the client/node.

I half wondered afterwards if slapping some crypto on top of it which would be mined by running the node and providing resources would help. My gut says easy yes, but my mind grimace at the abomination.

zoklet-enjoyer2• 2 years ago

We have proof of stake now. The nodes could be run by the chain validators and they get a cut of the staking rewards. Look up how proof of stake works on the Cosmos chain. You could totally do this and I bet it would take off, at least in that section of the Internet that's into Cosmos/Tendermint chains. I'd use it

ssijak3• 2 years ago

I was definitely thinking of some kind of proof of stake, not proof of work.

zoklet-enjoyer2• 2 years ago

Hahaha one downvote. I love to see it

worksonmine4• 2 years ago

but my mind grimace at the abomination

Why would that be an abomination? It's a perfect use-case. Like you noticed people need incentives to volunteer their hardware. If you hate crypto because it's crypto you can just use fiat instead.

komali23• 2 years ago

Like you noticed people need incentives to volunteer their hardware.

I wonder if this is because "volunteer your hardware" projects sometimes involve someone making else money, and if someone else is making money but not you, why should you donate your hardware?

For the truly libre "hardware donation" projects, they seem to be doing ok without financial incentivization. What immediately comes to mind is the petabytes of data flying around on peer to peer systems through torrenting. I know people that spend thousands of dollars a year on upkeep and upgrades for what are essentially super seedbox homelabs (I'm one of them too :P )

There's also communities like soulseek where people keep TBs of music up, often seeking out rare tracks to make available to the community for free.

There's folding@home and seti@home, and I'm sure other similar projects I haven't heard of, where people donate cycles just for the common good.

folding@home is a great example because we can directly compare the people that are "incentivized" to participate with bananocoin, a cryptocurrency rewarded based on work cycles in folding@home. You can see all bananocoin miners here under the banano.cc team: https://stats.foldingathome.org/ That team is in first place for work completed, however are only just surpassing the linus tech tips team, and not to mention compared to a bunch of other teams (and private "donors") they're a very small % of work completed for folding@home

So therefore I disagree that people "need" incentives, there just needs to be no, erm, disincentives, if that's a word.

bawolff• 2 years ago

I mean, how do you verify nodes are being honest and not just sending fake data for the free crypto (like what happened with seti@home and there wasn't even money involved)

Not to mention, where is the value of this coin going to come from? Will people pay to use this search engine? That seems unlikely.

It doesn't sound like the perfect use case to me.

lifty2• 2 years ago

Not sure why it would be an abomination. This is the exact use case which is a fit for cryptocurrency networks.

rakoo11• 2 years ago

You have to look beyond the surface. Cryptocurrencies work specifically to address a system where no node can trust any other node. If I cannot trust any other node, why would I fetch anyone else's index, or ask them for the results of a query, or even talk to them ?

Unless there can be a way to trivially verify what others tell you, crypto currencies are a dead end

6510• 2 years ago

After that the issue becomes ranking. Should say became since LLM's could both rank pages and generate them on "demand" to fit the query.

YaCy has so many buttons I'm not even sure if it lacks it but playing around with it it is very cool to crawl large amounts of pages and serve requests until you want to do other things with the computer and the background process is to bloated. Something like a turtle mode like torrent clients have would be useful.

Long ago there was a Chinese p2p client with a rootkit that would seed at 1 kb. I haven't used it but was told it worked remarkably well.

mdaniel3• 2 years ago

Nothing new under the sun, as they say: https://www.presearch.io/engine and just as you said I was unwilling to run a closed-source node binary

colinsane• 2 years ago

if the situation is really "nobody will run this software unless i pay them to", then you're doomed regardless. there's nothing wrong with the classic route: package your software for the stores/distros you're familiar with, make your software as easy to package as humanly possible for anyone else who'll come around, document the hell out of it, submit it to the handful of top-level news feeds from which it'll percolate, and then wait. maybe you don't like waiting?

rasulkireev• 2 years ago

Love it. Super easy to self host and use. Now I have a personal Google!

maxloh• 2 years ago

See also: Presearch, another decentralized search engine, claimed that it will be open source. No source code available at the moment though.

https://presearch.com/

b2bsaas00• 2 years ago

Could this be used for a Torrent search engine?

fddrdplktrew• 2 years ago

if it is not censored, probably?

worksonmine4• 2 years ago

Recently there was a distributed tracker on the front page. Probably more what you're looking for.

BLKNSLVR• 2 years ago

Bit Magnet: https://bitmagnet.io

rakoo11• 2 years ago

Note that it's not a distributed tracker, it's an indexer/tracker/search engine that uses distributed resources (the nodes in the dht)

feverzsj• 2 years ago

btdig is still alive.

qingcharles• 2 years ago

btdig has the data, but its search is subpar :(

vGPU• 2 years ago

Has it gotten any better recently?

I run a node but I haven’t actually used it as a search engine in a while, as I found the result quality to be exceedingly poor.

rahen• 2 years ago

I remember trying it for a while in 2012, but the results were essentially worthless, probably because there were so few nodes/crawlers back then. I guess the more users there are, the better the results.

viraptor• 2 years ago

Alternatively, ignore the public network (it's still useless) and run it as your own crawler. Seed it with your browsing history, some aggregators like HN, your favourite RSS feeds, etc. and you'll be good.

WarOnPrivacy3• 2 years ago

I remember trying it for a while in 2012, but the results were essentially worthless,

I had mine crawling gov, mil, etc sties for pages that Google was starting to delist back then. Inbound requests were heavy with porn until I tweaked - IDK, something.

Avamander• 2 years ago

No.

Either it picks up too much garbage if you allow any P2P data exchange (can't allow only outgoing AFAIK) or it kinda only knows about the sites you know about. Which kinda defeats the purpose.

Even assuming you just want a specific index for yourself of your own content then it struggles to display useful snippets about the results, which makes it really tedious to shift through the already poor results.

If you try to proactively blacklist garbage, which is incredibly tedious because there's no quick "delete from index and blocklist" button under index explorer, then you'll soon run into an unmanageable blocklist, the admin interface doesn't handle long lists well. At some point (around 160k blocked domains) Yacy just runs out of heap during startup trying to load it which makes the instance unusable.

It also can't really handle being reverse proxied (accessed securely by both the users and peers).

It also likes to completely deplete disk space or memory, so both have to be forcefully constrained. But that ends up with a nonfunctional instance you can't really manage. It also doesn't separate functionality enough that you could manually delete a corrupt index for example.

Running (z)grep on locally stored web archives works significantly better.

bobajeff• 2 years ago

Those are pretty bad issues. I remember using it along time ago and only remember the results being bad. I've heard that Yacy could be good for searching sites you've already visited but it sounds like even that might not be a good use case for it.

I do understand the taking up of disk space thing. It's hard to store text of all your sites without it talking up a lot of space unless you can intelligently determine which text is unique and desired. Unless you are just crawling static pages it becomes hard to know what needs to be saved or updated.

RGBCube• 2 years ago

    curl failed to verify the legitimacy of the server and therefore could not
    establish a secure connection to it. To learn more about this situation and
    how to fix it, please visit the web page mentioned above.

Can't seem to access the page.

gonesilent• 2 years ago

Infrasearch / Gonesilent sold to Sun turned into project JXTA and died.

mdaniel3• 2 years ago

While trying to read more about it, turns out there's an O'Reilly book, too: https://www.oreilly.com/library/view/jxta-in-a/059600236X/ch... and there's also this https://wiki.wireshark.org/JXTA (I'm guessing those specification links are in wayback but I didn't chase them)

charcircuit• 2 years ago

Are the results still being gamed by sites using content keyword stuffing? The last time I used it the searching and ranking technology felt like they were 40 years behind state of the art.

liotier• 2 years ago

In distributed indexing, spam management seems a much bigger problem than the indexing itself.

boyter• 2 years ago

I actually half wrote a RFC of a spec and 2 implementations of a federated search last year. Rather than do the disturbed hash table that yacy does.

I wanted results to be re-rankable by the peers by sharing the scores that went into them. The idea being with a common protocol based on the ideas of ActivityPub you could get peers of searches working together to hopefully surface interesting things.

Something I should probably finish and publish at some point. It worked to the hundreds of peers I tested.

The reason I mention this is because I wanted to also add a front into yacy which tuned out to be harder than I expected. It’s a wonderful project and you can find great stuff through it but the way the peers return results sometimes it’s hard to find it again. It’s also not quite as hackable as I would have hoped at the time probably due to he project age.

I still think there is value in it though and I’d love to see yacy have its protocol explained as an apex so people could,build implementations in other languages more easily.

detourdog• 2 years ago

I remember the first days of gopher browsing were like that. Gopher browsing to me was like swinging on vine to vine. The trick was remembering/documenting where each vine went.

arboles2• 2 years ago

Sort of hijacking the thread to ask, can YaCy or similar, be an alternative to Google's Programmable Search Engine? All I use it for is limit a search to a medium-sized list of domains. The aspect that makes running a search engine difficult on your own is lack of resources for crawling, I expect. But since I only care about a small list of domains, could I ditch Google's and run my own crawler like YaCy?

gtirloni• 2 years ago

Is that the deceased code search tool?

You could run Sourcegraph and import/sync those repositories.

Or you could run your own ElasticSearch/Melisearch and crawl the websites yourself (if you're interested in things other than git repositories).

arboles2• 2 years ago

Is that the deceased code search tool?

No, it's Programmable. Though it's not actually programmable. I should've written Custom Search Engine instead, that's also a name for it.

cse.google.com - It's quaint that past the modern landing page, when using the search portal today, you still get some outdated iteration of Google UI design.

It's used, for example, for making OSINT searches.[0] Or at some point by at least one Wikipedia editor for a custom list of Reliable Sources for Anime & Manga.[1]

[0] https://www.osintme.com/index.php/2020/09/28/

[1] https://gwern.net/me#wikis

anthk2• 2 years ago

Ugh, Java. I'll wait for something like i2pd does for I2P, something called yacyd either in c, c++ or golang.

ravenstine• 2 years ago

What's your objection to Java?

anthk2• 2 years ago

High CPU and RAM usage.

WarOnPrivacy3• 2 years ago

Yacy's still around. Nice.

After a year or two of hosting a Yacy instance (2014?) I started winding up on some general (probes, etc) blacklists.

I also host a small mail server and I was getting mail returned. I'd force an IP swap and a few weeks later it'd be the same. I had to let Yacy go.

1oooqooq• 2 years ago

So that is how they block a people's search/crawler. Didn't thought they would use the most complicated method.

They also use block lists to add every single TOR node (even if not an exit) and every VPN under the sun (except for streaming, because, why would them, that's why they exist)

WarOnPrivacy3• 2 years ago

So that is how they block a people's search/crawler.

It seems less that was the intent and more they blacklist IPs with behavior they find annoying. They're super general lists.

They also use block lists to add every single TOR node

This annoyed the crap out of me. The stupid Dan list guy made it as easy as he could to lump low risk bridges in with high risk exit nodes.

renegat0x0• 2 years ago

There are already many project about search:

- https://www.marginalia.nu/

- https://searchmysite.net/

- https://lucene.apache.org/

- elastic search

- https://presearch.com/

- https://stract.com/

- https://wiby.me/

I think that all project are fun. I would like to see one succeeding at reaching mainstream level of attention.

I have also been gathering links meta data for some time. Maybe I will use them to feed any eventual self hosted search engine, or language model, if I decide to experiment with that.

- domains for seed https://github.com/rumca-js/Internet-Places-Database

- bookmarks seed https://github.com/rumca-js/RSS-Link-Database

- links for year https://github.com/rumca-js/RSS-Link-Database-2024

fsflover• 2 years ago

But which of those projects are distributed and FLOSS?

legrande• 2 years ago

Also these:

https://swisscows.com/en

https://search.disconnect.me/

https://www.ecosia.org/

https://metager.org/

https://searx.space/

ColinHayhurst• 2 years ago

https://www.mojeek.com/ self-disclosure, mojeek team member

wongarsu• 2 years ago

To be fair, of those only Apache Lucene predates YaCy. YaCy is very mature, but in terms of relative popularity for general web search probably peaked around 15 years ago.

buffalobuffalo2• 2 years ago

I ran YaCy for a while, but not as a node on their distributed search index. I just ran it as a search engine for all my own bookmarks. Unfortunately I never found a particularly good way of getting bookmarks into the system. So eventually I shut it down. Cool idea in theory though.

justusthane• 2 years ago

I have plan that I haven’t implemented yet, but I want to route all my outbound internet traffic through a Squid reverse proxy, which will in turn add every visited URL to YaCy (except for domains I choose to exempt).

That way I’ll have a fully searchable index of every website I ever visit, which will hopefully solve the “Oh shit, what was that one website I found about X two months ago?”

A potentially easier thing to do would be create a bookmarklet that adds the current page to YaCy.

buffalobuffalo2• 2 years ago

Yeah. Bookmark indexing was my original goal. But yacy doesn't have a great interface for that. Doable with some work, but not something i wanted to sink too much time into.

mdaniel3• 2 years ago

relevant: https://github.com/ArchiveBox/ArchiveBox#readme and https://github.com/Rhizome-Conifer/conifer#readme (nee "webrecorder/webrecorder")

fortran77• 2 years ago

Related to this — I’d love to see individuals making web pages again, and federated search engines indexing them. People don’t make their own hobby or fan or art websites anymore, and I think that’s partly because nobody will every find them with the big search engines.

emrah• 2 years ago

I think it would be nice if the search results were "distributed" rather than deterministic.

So when i enter the same keywords, let's say there are 50 pages each of which would be equivalently good result for the search, rather than one page "winning", the search engine would alternate the winner among the many possibilities

jrussbowman2• 2 years ago

Nice to see search projects are still popping up. After a move, family life taking over and me getting more interested in Unreal Engine, my poor search engine is now more of an experiment in seeing how well it runs while basically on life-support maintenance updates I do. Starting to think I honestly should just take it down and save my $50 a month I spend maintaining it.

But I'll post it in a hacker news comment and maybe you all will give it enough traffic I can get excited about it again, lol

https://www.unscatter.com

jrussbowman2• 2 years ago

And for my immature moment of the day, the above comment was comment #69

fho• 2 years ago

I've been using several times over the last decades and never got good results. I think one instance is still running on my old computer at uni :-)

dredmorbius• 2 years ago

Previously:

YaCy – your own search engine | https://news.ycombinator.com/item?id=32597309 | 2 years ago | 93 comments

YaCy: Decentralized Web Search | https://news.ycombinator.com/item?id=22246732 | 4 years ago | 41 comments

YaCy – The Peer to Peer Search Engine | https://news.ycombinator.com/item?id=17089240 | 6 years ago | 3 comments

YaCy: a free distributed search engine | https://news.ycombinator.com/item?id=12433010 | 8 years ago | 24 comments

YaCy: Decentralized Web Search | https://news.ycombinator.com/item?id=8746883 | 9 years ago | 29 comments

YaCy takes on Google with open source search engine | https://news.ycombinator.com/item?id=3288586 | 12 years ago | 17 comments

treprinum• 2 years ago

Is it worth dedicating 1-2 low power NUCs (4-8 core) to this on a 250MBit/s connection? Or does it need beefier CPUs/network?

nairboon• 2 years ago

If you run YaCy with docker and it is still a junior peer, does the search return results from the global index or just the one that appears to be 'preinstalled'?

YaCy, a distributed Web Search Engine, based on a peer-to-peer network

Related Stories