asciimoo

I like the idea. Could you share your RepoSearch app? Also, we have Discord & IRC channels, please join and start brainstorming.

I agree with you, but Bun was still something of a flagship project for Zig. It received a lot of attention, which indirectly helped Zig's popularity.

I was wrong to be upset this whole time that the rewrite would hurt Zig. This is one of those rare occasions when I’m glad I was wrong. Interesting insights.

We need projects like this. Automatically classifying the files is smart.

I'm working on a similar application called Hister (https://github.com/asciimoo/hister). I should borrow some of your ideas. =]

Multiple users can use a shared instance and collect their indexed content in a central place. Hister has user handling and a "public mode" as well: https://hister.org/posts/public-search

Thanks for the kind words =]

There is already an ongoing discussion about the topic: https://github.com/asciimoo/hister/issues/387

The currently discussed solution relies on the browser extension, but mobile Firefox has extension support.

I accept donations ;)

Hister has a built in crawler with standard HTTP lib and browser based backends, you can feed your link collection to it. Also, Hister supports importing your existing browser history automatically using either of the mentioned backends.

Hister stores only the text content of HTML/pdf pages. 1000 documents require around 80-100MB of storage and there is still plenty of room to optimize for storage space.

I'm using it for 6-7 months and my index size is below 1GB with almost 10k pages.

Also, a downside of the proxy approach: it does not handle properly JS based websites and cannot identify dynamic content changes. Our extension periodically checks if the browser tabs' content has been changed and automatically updates the index when change detected.

It is possible. The automatic website indexing can be turned off in the extension and manual indexing can be triggered via the command line tool, the extension popoup or by hotkeys.

What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful?

There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.

Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?

My long term goal is exactly this. I'd like to add federation/P2P feature [2][3] to evolve from being a private search companion. I'd appreciate any help designing the system.

[1] https://blevesearch.com/docs/Home/ [2] https://github.com/asciimoo/hister/discussions/432 [3] https://hister.org/posts/public-search

Unfortunately mobile Chrome browsers don't support browser extensions, but our extension works well on mobile Firefox.

Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found.

I agree, we should add more extractors [1]. Can you recommend extractors you missed?

[1] https://github.com/asciimoo/hister/issues/305

Absolutely, this is a great example where Hister can shine.

I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.

In its current form it cannot handle this amount of data efficiently (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.

Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers. Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/

Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister (https://hister.org/).

Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.

Take a look at how the MCP can be utilized: https://hister.org/posts/give-your-ai-assistant-a-private-me...

I'm still working on a self-hosted search service called Hister with the goal to reduce dependence on online search engines.

Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. It provides offline result previews, a flexible web (and terminal) search interface & query language to explore saved content with ease or quickly fall back to traditional search engines.

I've been using it for a few months and as my local index is growing I can avoid opening google/duckduckgo/kagi - and even websites listed in results - more and more frequently.

The initial reception is overwhelmingly positive with already more than 30 contributors and hundreds of contributions - perhaps you can find it useful as well. (Or at least have some constructive criticism =])

GitHub: https://github.com/asciimoo/hister

Website: https://hister.org/

Small read-only demo: https://demo.hister.org/

I'm working on a self-hosted search service called Hister (https://hister.org/ - https://github.com/asciimoo/hister) with the goal to reduce dependence on online search engines and AI answers.

Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. It provides a flexible web (and terminal) search interface with offline result previews & detailed query language to explore collected content or quickly fall back to traditional search engines.

It can provide a privacy-respecting search experience for serving "recall" type searches where users retrieve previously visited content, but falls short in "discovery" type searches (yet).

As a search/metasearch developer, I can heavily recommend Uruky. It is by far the best third party search alternative in terms of how they approach privacy and transparency. Keep up the good work Bruno! <3

Tons of new features & enhancements. Upgrade your instances.

  Location: Europe, Budapest
  Remote: Yes 
  Willing to relocate: No
  Technologies: Go, Python, svelte, search engines, scrapers, crawlers, data gathering, privacy, free software
  Résumé/CV: https://github.com/asciimoo
  Email: asciimoo+hnjob@gmail.com

~20 years of development experience. Author of many somewhat successful free software projects like Searx, and Go-Colly.

I'm seeing the same increased activity around my search engine project (https://github.com/asciimoo/hister). While Google's decision is very controversial, it's good to see that people are seeking for alternatives - nice motivation boost to keep developing alternative search projects.

Nice! Hister provides an MCP interface so you can easily integrate it to your RAG setup.

Yes, the extension extracts the content of the tabs as your browsers renders them. This includes applying all your extension's modifications as well.

Thank you for your feedback, it's super useful to get insights from users.

I agree, browser import has rough edges. The issue you mentioned is known: https://github.com/asciimoo/hister/issues/31 . I try to prioritize it and find time to fix it.

This is an awesome idea in theory, I'd love to go to this direction, but it's a surprisingly complex topic. I find it hard to come up with an implementation that can guarantee both result quality (no malicious actors) and user privacy.

I'd appreciate any kind of help designing such system. We are on IRC/Discord/Github/Codeberg.

If I wanted to use use my domain list to start hister, to download my preconfigured / like domains?

Yes, Hister has a built-in crawler which supports standard HTTP and different browser based backends

Can I make some pages to rank higher in it?

You can create priority rules to boost the ranking of the matching domains/URLs

Can I assign tags to pages (by which I could later on filter?)

It is possible to add a label to indexed documents

It isn't supported yet, but it's a good idea and quite simple to implement it. I've added it to the issue tracker: https://github.com/asciimoo/hister/issues/431 . Thanks for the suggestion.

Ohi, I'm the author of the open source Searx metasearch engine.

I'm working on a self-hosted search service called Hister with the same goal when I started Searx development: reduce dependence on online search engines.

Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. It provides a flexible web (and terminal) search interface & query language to explore saved content with ease or quickly fall back to traditional search engines. This is a fundamentally different approach than what Searx follows and solves most of the weaknesses of metasearch engines. Of course it has its own weaknesses as well, but most of these are not conceptual and can be resolved by improving the software (and datasets)

I've been using it for a few months and as my local index is growing I can avoid relying on external search engines - and even websites listed in results - more and more frequently.

The initial reception is overwhelmingly positive with already more than 30 contributors and hundreds of contributions. Currently it can help with "recall" type searches mainly, but I'm planning to provide pre-indexed thematic datasets and I'm drafting a peer-to-peer index sharing concept. Maybe you can find it useful as well (or at least have some constructive criticism =]).

Links: - https://hister.org/ - https://github.com/asciimoo/hister - Background/motivation/beginnings: https://hister.org/posts/how-i-cut-my-google-search-dependen... - Small read-only demo: https://demo.hister.org/

Ohi, author here! Feel free to A.M.A.