lorey

Their response:

The team that made dataroom has stated that they did not use any of papermark’s code and that dataroom was made from scratch with inspiration from existing document sharing softwares, and that this post’s allegations of us stealing code are false. [...]

The screenshots clearly show they copied whole pages verbatim, both design and texts. The founder, Nico Laqua, basically responding with "we didn't copy _code_" and not taking any responsibility says a lot about his and his company's moral code. It might not be enough to get sued. That doesn't make it right.

https://x.com/nico_laqua/status/2070158170937581951

That is not what the article argues.

Haha, very true. Exactly as described in the article.

This is true with one caveat.

In most cases, e.g. with regular ML, evals are easy and not doing them results in inferior performance. With LLMs, especially frontier LLMs, this has flipped. Not doing them will likely give you alight performance and at the same time proper benchmarks are tricky to implement.

This is a very good point. When I came in, the founder did a lot of evaluation based on a few prompts and with manual evaluation, exactly as described. Showing the results helped me underline the fact that "works for me" (tm) does not match the actual data in many cases.

Doesn't this depend a lot on private vs company usage? There's no way I could spend more than a few hundreds alone, but when you run prompts on 1M entities in some corporate use case, this will incur costs, no matter how cheap the model usage.

It's not you, it's the HN hug of death. There's so much load on the server, I'm barely able to download the redis image I need for caching...

Thanks. Will take a look.

Depends on your remaining budget ;)

I've skipped that in the article, but absolutely!

Fixed, thanks. Not a native speaker.

That's interesting. Similarly, we found out that for very simple tasks the older Haiku models are interesting as they're cheaper than the latest Haiku models and often perform equally well.

Pushed a fix. Could you check, please?

Any resources you can recommend to properly tackle this going forward?

Will fix, thanks :)

Totally agree with your point. While I can't say specifically, it's a traditional (German) business he's doing vertically integrated with AI. Customer support is really bad in this traditional niche and by leveraging AI on top of doing the support himself 24/7, he was able to make it his competitive edge.

This went straight to prod, even earlier than I'd opted for. What do you mean?

Appreciate the feedback, will work on that.

You're right. We did a few use cases and I have to admit that while customer service is easiest to explain, its where I'd also not choose the cheapest model for said reasons.

Yes, absolutely. This aligns with what we found. It seems to be necessary to be very clear on scoring (at least for Opus 4.5).

Very interesting points. Would you mind sharing a few examples of when cherry-picking is necessary and why atomic changes are a lie?

I'm using a monorepo for my company across 3+ products and so far we're deploying from stable release to stable release without any issues.

Yes, there's the <link rel="icon"...> an other specific tags providing far more options than the old favicons.

For anyone interested: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/...

Thanks, this is how I feel about this, too.

Spent the last three months building a competitor/lookalike ML model + API. Started using plain embedding similarity and quickly realized you end up with similar noisy results as ocean.io. Ended up using similarity learning which works quite well with little data. Launched this as an API and small web app. Hardest part right now is to fend off scrapers honestly.

Examples:

- YC: https://markets.apistemic.com/companies/y-combinator-goaq9

- uber: https://markets.apistemic.com/companies/uber-com-ojj2j

- Anthropic: https://markets.apistemic.com/companies/anthropicresearch-yx...

Try it for any company here: https://markets.apistemic.com

I don't understand why so many VCs fall for "not invented here". Vibe-coded or not, this is just another in-house solution, inferior and more expensive than most out-of-the-box products already out there.

Had to modify the title slightly to pass validation. Also "How [I algo...]" got removed, did not know that happens automatically.

Why GPT-based then? There are libraries that do this: You give examples, they generate the rules for you and give you a scraper object that takes any html and returns the scraped data.

Mine: https://github.com/lorey/mlscraper Another: https://github.com/alirezamika/autoscraper

<div class="hidden">Actual name: Batman</div>

Most explicit CSS rules allow you to spot this, implicit rules won't and possibly can't.

Personally, this feels like the direction scraping should move into. From defining how to extract, to defining what to extract. But we're nowhere near that (yet).

A few other thoughts from someone who did his best to implement something similar:

1) I'm afraid this is not even close to cost-effective yet. One CSS rule vs. a whole LLM. A first step could be moving the LLM to the client side, reducing costs and latency.

2) As with every other LLM-based approach so far, this will just hallucinate results if it's not able to scrape the desired information.

3) I feel that providing the model with a few examples could be highly beneficial, e.g. /person1.html -> name: Peter, /person2.html -> name: Janet. When doing this, I tried my best at defining meaningful interfaces.

4) Scraping has more edge-cases than one can imagine. One example being nested lists or dicts or mixes thereof. See the test cases in my repo. This is where many libraries/services already fail.

If anyone wants to check out my (statistical) attempt to automatically build a scraper by defining just the desired results: https://github.com/lorey/mlscraper

Isn't this what contracts were created for?

Roses are red, their logo is blue. They scrape your data, but sue if you do.