mikkel125

Are all URLs I visit sent to your server? Or is there a way I can see related HN/Twitter comments only on a button click when I choose so?

I don't use twitter, but would love to see related reddit threads.

That's a good point. "Long-term" may be overly optimistic. A 6-12mo companion would already be great.

Yes, and that's where I started looking: Upwork, Freelancer.com, Malt, Lemon.io. However, I find it hard to find the "jack of all trades" type of character I'm looking for. Freelancers nowadays are specialized in a small subset of technologies where they enjoy the most working on. It often does not go well with the quick&dirty hacker image that I'm looking for.

But maybe there's some place where this type of hackers hang out/advertize themselves?

How can I contact you?

give them a salary

That's more tricky if the freelancer is not a resident from the same country as my company's headquarters.

I guess tailored freelance contracts can always be crafted: monthly fixed rate, span over 12-24mo, 4-week termination notice... so that it feels salary-like. That's not an issue. The real pain is where to find the type of freelancer accepting these and with the correct mindset.

I read from other comments that you're writing your crawler bot yourself. Instead of crawling "from scratch", have you considered using an existing DB like Commoncrawl? Or is there something else that you index not present in Commoncrawl?