HN user

lpellis

488 karma

Let AI proofread your site. https://pagewatch.ai

Posts11
Comments148
View on HN

https://pagewatch.ai/ - make sure LLM's can understand your site.

There is a surprising amount of edge cases that can cause ChatGPT or others to misunderstand your pages. Some models can handle div based tables, some want alt tags but cannot understand title tags, etc.

I built the tool to check your site as close as possible to what a human would see and then compare it with LLM's.

It was a weird journey trying to tease this info out of the models, they will happily lie, skip checking sites or just make things up.

A task framework could be very useful, setting up celery task can get complicated very fast, especially once you have multiple servers with rolling deploys or recurring task (celery-beat). Rails has active-job and is always something I felt was missing from Django.

It is cloudflare that blocked us, we scrape using a regular chrome instance (with the latest user-agent), but cloudflare are sometimes very aggressive in their blocking. I'm busy adding a proxy option to handle these cases automatically, and some way to whitelist our scraper.

Seems like a fun scraping project, I think you have to work on extracting more accurate categories though, for example this link does not really include snowboards for me: https://www.searchagora.com/search?query=Snowboard And the first products I clicked have rather weird descriptions, https://www.searchagora.com/products/snowboard-bd2a90aa-6808...

Maybe its my location (South Africa) but I also cannot visit the product store when I click through

Surprising how well I can see the image even after putting the separation to 64. The image is just a few thin strips yet I can easily see the whole thing if I follow the scrolling, really cool.