How to build a search engine with common Unix tools (2018) [pdf]

https://www.iaria.org/conferences2018/filesDBKDA18/AndreasSchmidt_Tutorial_SearchEngine.pdf

by faizshah • 6 years ago

197 26 6 years ago

The linked site on the slides is password protected, and internet archive is silent on it as well; Does anyone have a copy of the referenced materials?

cr0sh2• 6 years ago

I've put out a request to his email address for access; if I hear anything, I will post back to this thread...

cr0sh2• 6 years ago

Should be open now:

https://www.smiffy.de/dbkda-2018/

ninjamayo• 6 years ago

why?

faizshah3• 6 years ago

There was a post earlier on using command line tools instead of Hadoop for quick data processing. This shows a non-trivial example of how you could implement a complex data pipeline and an overview of some of the commands you could learn if you’re interested.

crmrc114• 6 years ago

Yeah, this is a pretty cool post I never considered doing something like this in the shell. It seems silly to me that I forget how powerful basic tooling in the native shell can be.. sometimes I have a jackhammer and I forget that a sledgehammer will do the job just fine.

pmlnr• 6 years ago

Because there's a big trend towards overengineering. I checked the Azure search solution, and my god it's complex. I ended up doing it in sqlite, and it perfect for a small fts document search engine. The article is similar: try to keep it simple; there are barely any cases where you'll use the real benefits of something lkke Azure search.

shakna• 6 years ago

I ended up doing it in sqlite, and it perfect for a small fts document search engine.

It even has it as a feature!

    CREATE VIRTUAL TABLE something USING fts5(x, y, z)

riddleronroof• 6 years ago

In SQL https://gist.github.com/sanealytics/0e910380576fbe4825455264...

pstuart• 6 years ago

sqlite has full text search capability: https://sqlite.org/fts5.html#overview_of_fts5

mickael-kerjean• 6 years ago

I did build something like this (but even simpler) for the support website of my side project: https://support.filestash.app/. a PHP script calling grep and displaying the results. It's very hacky but is good enough for its intended use case: search through the entire IRC chat log

stevekemp• 6 years ago

I dropped you a mail about your search form :)

probably_wrong• 6 years ago

I did something similar too. For my last move, I wrote a detailed list of which item was in which box. My original plan was to add a QR code to each box, so I could quickly see what's inside.

But once I was done, I realized that I had it backwards. Therefore I wrote a PHP page to grep the list, and figure out in which box a specific item was.

blondin• 6 years ago

man... slides always feel out of context for me. i would rather a blog post than slides. these seem to cover the basic theory as well...

heinrichhartman3• 6 years ago

Those methods were always appealing to me, and I tried to bild some home use applications with it (document management, even some bash cgi scripts) but they fall over quite soon:

* How do you deal with words like "---" in your text that look like the match separator of grep?

* What if your filenames contain spaces?

* Are the sed/awk/perl one liners really all that readable and correct?

* How to catch and report failure conditions ... in pipe steps?

This stuff is great for interactive use and one-off ETL, not for applications.

Not sure what real alternatives are that give you:

- parallel execution

- seamless composition (like |)

- object passing not byte streams

- Quick to write.

Most of the time I switch to Python for this, but it does not give you sane parallelity. Sure you can do this with Java + Akka, but this takes days to build out...

Any recommendations?

bkq• 6 years ago

How do you deal with words like "---" in your text that look like the match separator of grep?

For this one you can use the "--" flag to signal that everything else should be treated as an argument.

    $ grep -rn -- -

heinrichhartman3• 6 years ago

yeah, I should have made that more clar: I was talking about the sed expression on p17. It looks for "--" in stdin. This could alsp be a word. I realize now that the first tr before removes all non a-zA-Z characters, so in this case it should not be an issue.

However intermixing text with separators is not trivial. There are reasons we use JSON/XML for exchanging structured data.

faizshah3• 6 years ago

The two I reach for first: Dask and SparkSQL

Dask is super easy and quick to learn provides similar features to spark but can be somewhat easier for the Pandas crowd. There's also Modin/Ray for this but I haven't tried it yet.

For very fast processing and ease of writing SparkSQL is the tool I reach for. Start a single node spark instance (super easy) then interactively wrangle ur data declaratively with SQL. Great for quick and dirty cleaning and aggregation of big-ish data.

If you're into google cloud BigQuery is currently my top tool for quick and dirty processing but u can do a lot more with ur 5$/1TB with a giant compute engine high mem instance and Dask or SparkSQL.

heinrichhartman3• 6 years ago

Thanks for this. I did not know about Dask! wow this looks great. Love the web-based task visualizations: https://distributed.dask.org/en/latest/web.html

duggan• 6 years ago

I always enjoy demonstrating various combinations of cat, grep, uniq, sort, and cut to folks unfamiliar with the command line; data scientists in particular.

Even if you can't ship a bash script to production, they're great tools for ad-hoc exploration and validation.

limesontoast2• 6 years ago

I am planning to read through this more thoroughly as something I really want is my own personal search engine. Ideally I need it to

1. store data locally for offline retrieval

2. Support indexing big sites including stack overflow, Wikipedia, Reddit, news.ycombinator.com, microsoft.com docs, and a bunch of other domains.

3. Be easy to add a single URL into the index from command line and optionally browser plugin. Only index that page, this would replace my bookmarks.

4. Optionally auto store browser history for a custom period of time, purge when expires.

Does anything like that exist?

ko56• 6 years ago

For those interested in building their own local offline datasets/search engines check out Kiwix and Zeal. Understand how they work.

Code is open and there are a ton of already created data dumps + indexes. You don't have spend time rebuilding a Wikipedia/wikidata/stackoverflow dump and index by yourself.

limesontoast2• 6 years ago

Thanks for the pointers, I'll check those out as well.

How to build a search engine with common Unix tools (2018) [pdf]

Related Stories