markpapadakis

What an inspired name :)

This looks like a disaster. It is like it was fast-tracked based on the oomph factor because seriously, how come you didn't notice how hard it is to read text or even notice overlapping objects/controls? Maybe once we use it for some time, we will all get it - it is possible - but as it now stands, I hope there will be an option to turn all that off, _especially_ on MacOS which is what I use to get work done.

https://github.com/skypjack/entt is a fantastic alternative; also, one of the most beautiful codebases/elegant designs I 've come across.

Velocity is about familiarity, about expressiveness, about being able to quickly translating what you want to accomplish to code. It’s also about, for me anyway, not being constrained by inherent limits of the language or runtimes. Like I said, checks all the boxes.

As for experience, I have been writing C since the very early nineties and moved on to c++ soon thereafter.

Again, all that is, for the most part anyway, not specific to this language.

All our infrastructure technology, services, and almost all “applications” we have built are written in C++. That’s millions of lines of code and span everything from IR, distributed storage and computation, ads serving, relational and columnar datastores, caching and much more.

It’s about velocity and experience. Language rarely matters ( and when it does, you can use what’s appropriate ). For us, C++ checks all the boxes.

We have been running hundreds of services that effectively consume from various topics, perform mostly simple transformations and produce back to other topics. Providing this kind of functionality makes a lot of sense.

We are using TANK, and that functionality is not available there, but given the benefits such a feature provides, we will likely use RP as well in order to save on development time and number of distinct services we use for various events streams transformations.

This is a great related podcast episode: https://maximumfun.org/episodes/sawbones/havana-syndrome/

Apologies for linking to something I wrote; I believe convenience is addictive: https://markpapadakis.medium.com/convenience-is-key-2aad97d5...

This works great for me; what's most important is understanding what it is and how it materialises. Dealing with it becomes much simpler once you can reason about it: https://markpapadakis.medium.com/how-to-prevent-burnout-and-...

This is quite ridiculous. I’d say it’s obviously ridiculous but others may have a different point of view. The government there is eager to side with Murdoch and his friends for favorable coverage so that they can pocket money from this extortion stunt while disregarding the impact to its many many citizens. Suggesting Bing is good enough is stupid because if their citizens were to use it over google if google quits that market, what then ? Government goes after them and if it does what if Microsoft also gets out? And if government doesn’t then doesn’t that show to the world that it was personal with google? People shouldn’t be stupid. It’s a low bar to clear.

This is fantastic news. What would you say were the hardest aspects of the design and implementation, in terms of effort and thinking that went into those accomplishments ?

https://medium.com/@markpapadakis/interesting-codebases-159f...

In addition to those I highly recommend studying ClickHouse’s codebase. There are brilliant design and engineering bits everywhere. I learned more from studying this codebase than most other I can think of, especially with regards to templates meta-programming ( I learned about “template template parameters” from coming across extensive use of those there ). It’s actually somewhat challenging to grok what’s going on — but it is worth pushing through until you “get it”.

Pocket ( https://getpocket.com/ ). It’s really bad. Been using it for years and it’s always been broken. Crashes often, narration sometimes works most of the time it doesn’t, other times it works if you force stop the app and restart it. It’s very slow too. I depend on for my long daily commutes and I only stick to it because InstaPaper is also very broken and there is no reasonable way to move the stories from pocket to instapaper. I am convinced the people who build it never really use it for those issues are otherwise trivial to encounter and I would think trivia to fix as well. I hope someone will build a better such app and make them irrelevant.

Postfix

kuro5hin, slashdot as it existed back when, dejanews, and s bunch of video gaming sites that went extinct

I never had coffee. I drink Diet Coke and I enjoy tea ( I have some once every week or so ). I have been writing code for over 25 years now, and I never felt that I was missing out but then again not sure I’d know either way.

From https://500ish.com/offline-curious-85363c4aed18

One trend I’m most curious about right now is the rise of no or low alcohol drinks. I noticed this in Europe years ago — that nearly everywhere you went they not only offered, but touted non-alcoholic beverages in bars — highlighting the opposite approach the US was taking: pushing more extreme alcoholic drinks like Four Loko and the like, with non-alcoholic drinks being almost a taboo here. That is changing, quite quickly, it seems.

Our (https://bestprice.gr/) services/“programs” generate three different types of events:

- Short events (no longer than ~1k in size) where the cost to generate and transmit is very low(sent to dedicated service via UDP). We can generate dozens of them before we need to care about the cost to do so. They are binary-encoded. The service that receives those datagrams generates JSON representations of those events and forwards them to all connected clients(firehose) and also publishes them to a special TANK partition.

- Events of arbitrary size and complexity, often JSON encoded but not always -- they are published to TANK(https://github.com/phaistos-networks/TANK) partitions

- Timing samples. We capture timing traces for requests that may take longer-than-expected time to be processed, and random samples from various requests for other reasons. They capture the full context of a request(complete with annotations, hierarchies, etc). Those are also persisted to TANK topics

So, effectively, everything’s available on TANK. This has all sorts of benefits. Some of them include:

- We can, and, have all sort of consumers who process those generates events, looking for interesting/unexpected events and reacting to them (e.g notifying whoever needs to know about them, etc)

- It’s trivial to query those TANK topics using `tank-cli`, like so:

  tank-cli -b <tank-endpoint> -t apache/0 get -T T-2h+15m -f "GET /search"  | grep -a Googlebot

All told, we are very happy with our setup, and if we were to start over, we’d do it the same way again.

This is what I believe is the the main cause of burnout and how to efficiently prevent it and deal with it if necessary:

https://medium.com/@markpapadakis/how-to-prevent-burnout-and...

I like how the blog post title is, likely, based on Google's Page and Brin seminal paper "The Anatomy of a Large-Scale Hypertextual Web Search Engine". :)

This blog post, and other in the series, mention RocksDB is used for the index, but it's not explicitly described how and to what end. I 'd love to know the details.

For a few million queries we needed to index, many of them close to 30 characters in length(some even longer than that), the generated index size for max edit distance 3, was really large.

So we used 3 indices (one for unigrams, bigrams, and trigrams respectively) -- and during query processing we would segment the input query and for each segment we ‘d consult any of those 3 indices that made sense and would keep the top-K ones, and then we ‘d proceed to the next segment and we ‘d consider the n-gram sequence of the “suffix” and “prefix” matches between the “carried” top-K from the previous segment and the current/next segment, and so on, until we have exhausted the query. Segmentation was particularly hard to get right.

It was a fairly involved process but it was what worked for us -- again, SymSpell is _very_ fast for short tokens, but we had to execute maybe 1000s of such SymSpell queries when processing an input query and it adds up quickly. (We will probably open-source our SymSpell implementation soon).

Got it. This is by design - though I suspect we need to dial down the cost for misspelled first character in prefixes.

We built our own FST, which is somewhat similar to the design of the Rust fst crate ( https://blog.burntsushi.net/transducers/#fst-construction ).

Could you please tell me what you entered?

As for prefixes and typos, in our case, the FST is used for both matching a prefix and spelling correction of the prefix.

The first character is treated specially; the chance of getting the first character wrong is very low(according to our findings) so only completions/corrections that score very high get to be matched.

SymSpell is great for tiny datasets(based on edit distance/operations, and dictionary size). It’s fast, but the generated index will likely be very large; furthermore, you can’t practically encode rules and transition costs (e.g cost from “f” to “ph” being zero for getting better results for [ifone]).

An FST is the better choice. We (https://www.bestprice.gr/) used SymSpell in the past, but have since switched to an FST based design, where the FST is used to match prefixes. We also store all queries encoded in the FST, together with their frequency, sorted by the query.

For each distinct corrected prefix, we determine the range in that (query, frequency) list using binary search, and then just iterate in that list from [first, last] in that range, and compute a score based on a language model and the edit cost(prefix to corrected prefix) (i.e noisy channel).

We encode many different rules in the FST including language domain ones, and building the index takes a few seconds for millions of queries. The language model is based on 5-grams and Stupid Backoff.

What does each of those 200 dimensions represent/encode?

This is a very interesting and pragmatic design. I tried a couple of queries and the results were great for most of them, but not so much for esoteric ones, or for queries involving names, which makes sense according to the implementation specifics outlined in their blog posts.

I never heard of this company before, and I am very interested in IR technologies( https://github.com/phaistos-networks/Trinity ). I am glad they are trying and it looks like they have a chance for something great there.

Ahrefs.com is also working on a large scale web search engine, but other than their CEO announcing it on Twitter, there hasn’t been any other update since.

Was Gumbo used for parsing crawled pages for indexing reasons ? How was it used internally ? I presume they use some headless chrome like technology for most of that nowadays.

I dislike whitespace sensitive languages or definition formats with a passion. Especially when tabs and spaces are not treated equally. I don’t mind python as much nowadays but YAML is borderline insulting to me. I hope we all move on to something more sane soon.

Thank you. Can you talk about the reasons that necessitated the rewrite from c++ to java? I didn’t realize Brin wrote any code that wasn’t python either.

Very curious about what that Java program is.

What's "Ganpati" ?