Ask HN: How have you optimized software?

https://news.ycombinator.com/item?id=32100184

by robalni • 4 years ago

2 2 4 years ago

news.ycombinator.com

Tell me about how you have made some software significantly faster (to run or compile).

What type of software was it? Why was it slow and how did you find the cause/solution? What did you do and how much faster did it get?

PaulHoule• 4 years ago

My favorite case is when me and my co-workers developed a SPARQL query that, given a topic in Freebase, would return other similar topics based on "other topics that have similar attributes". For instance, two people would get more points if they were born in the same country, or if they both played the guitar, etc.

We estimated that it would take 100 years to run this query for all the topics in our database, which wasn't acceptable.

I figured out that most of the computational effort was in dealing with attributes that were shared by a large number of topics (such as "is a person") and since we were doing something like tfidf scoring the actual contribution these attributes made to the score was small. So we could eliminate almost of the work by throwing out attributes that were shared by too many topics without hurting the scores.

We also sped it up by organizing it as a batch job so we computed all the similarity scores at once and wrote a simple "map reduce" framework that mainly worked by sorting the data and breaking it up into hash buckets that could be processed with multiple threads on one machine.

This job took 20 minutes to run, which was a huge speedup!

stop50• 4 years ago

By using cached metadata and parallel http queries. It is already faster than the redhat official command for satellite.

Ask HN: How have you optimized software?

Related Stories