maerten

Nice article!

The second kind is nastier. They change things in a way that doesn't make your scraper fail. Instead the scraping continues as before, visiting all the links and scraping all the products.

I have found that it is best to split the task of scraping and parsing into separate processes. By saving the raw JSON or HTML, you can always go back and apply fixes to your parser.

I have built a similar system and website for the Netherlands, as part of my master's project: https://www.superprijsvergelijker.nl/

Most of the scraping in my project is done by doing simple HTTP calls to JSON apis. For some websites, a Playwright instance is used to get a valid session cookie and circumvent bot protection and captchas. The rest of the crawler/scraper, parsers and APIs are build using Haskell and run on AWS ECS. The website is NextJS.

The main challenge I have been trying to work on, is trying to link products from different supermarkets, so that you can list prices in a single view. See for example: https://www.superprijsvergelijker.nl/supermarkt-aanbieding/6...

It works for the most part, as long as at least one correct barcode number is provided for a product.

i just downloaded the app on iPhone and the dive planner allows you to change the gas mix (oxygen percentage) so it looks like it supports nitrox.

@dockimbel: perhaps adding some syntax coloring of your code example might help here, the syntax is hard to parse if you don't know what to look for..

Down indeed

We're using CasperJS (basically a simple interface to PhantomJS), works great for us. Have a look here: http://casperjs.org/

Pretty cool, although it keeps crashing my Safari on OSX Yosemite beta somehow :)

I wasn't a fan of glossy displays either, but the screen on the rMBP seems a lot less reflective than the previous generations' Macbook Pros (the ones with build-in DVD drive). The reflections are much darker, and kind of hard to notice.

Nice, although tmux -S /tmp/pairprog and tmux -S /tmp/pairprog attach isn't that hard to type :-)