Show HN: Crawl a modern website to a zip, serve the website from the zip

by unlog • 2 years ago

223 66 2 years ago

unlog16• 2 years ago

I'm a big fan of modern JavaScript frameworks, but I don't fancy SSR, so have been experimenting with crawling myself for uploading to hosts without having to do SSR. This is the result

mikeqq20244• 2 years ago

for a long crawling task, if exited/broken for any reason, does it save and restore at the next run?

eXpl0it3r• 2 years ago

The README says:

Can resume if process exit, save checkpoint every 250 urls

ryanwaldorf• 2 years ago

What's the benefit of this approach?

unlog16• 2 years ago

That the page HTML is indexable by search engines without having to render in the server. Such unzipping to a directory served by nginx. You may also use it for archiving purposes, or for having backups.

brudgers• 2 years ago

One possible advantage I see is it creates a 1:1 correspondence between a website and a file.

If what I care about is the website (and that's usually going to be the case), then there's a single familiar box containing all the messy details. I don't have to see all the files I want to ignore.

That might not be a benefit for you and not having used it, it is only a theoretical benefit in an unlikely future for me.

But just from the title of the post, I had a very clear piccture of the mechanism and it was not obvious why I would want to start with a different mechanism (barring ordinary issues with open source projects).

But that's me and your mileage may vary.

earleybird• 2 years ago

Understood that this is early times, are you considering a licence to release it under?

unlog16• 2 years ago

Sure, I forgot about that detail, what license do you suggest?

yodon2• 2 years ago

MIT and BSD seem to be by far the most common these days (I generally do MIT personally)

meiraleal4• 2 years ago

AGPL

yodon2• 2 years ago

+1 can you add a license file with an MIT or BSD or whatever your preference is? (It's very cool. I'd love to help with this project, I'm guessing others would as well)

jayemar• 2 years ago

Is the output similar to a web archive file (warc)?

unlog16• 2 years ago

That's something I haven't explored, sounds interesting. Right now, the zip file contains a mirror of the files found on the website when loaded in a browser. I've ended with a zip file by luck, as mirroring to the file system gives predictable problems with file/folder names.

toomuchtodo• 2 years ago

https://news.ycombinator.com/item?id=40628958

https://github.com/internetarchive/warctools

jll29• 2 years ago

Microsoft Interne Explorer (no, I'm not using it personally) had a file format called *.mht that could save a HTML page together with all the files referenced from it like inline images. I believe you could not store more than one page in one *.mht file, though, so your work could be seen as an extension.

Although UNIX philosophy posits that it's good to have many small files, I like your idea for its contribution to reduceing clutter (imagine running 'tree' in both scenarios) and also avoiding running out of inodes in some file systems (maybe less of a problem nowadays in general, not sure as I haven't generated millions of tiny files recently).

jdougan2• 2 years ago

.mht us alive and well. It is a MIME wrapper on the files and is generated by Chrome, Opera, and Edge's save option "Webpage as single file" and defaults to an extension of .mhtml.

When I last looked Firefox didn't support it natively but it was a requested feature.

rrr_oh_man• 2 years ago

When I last looked Firefox didn't support it natively but it was a requested feature.

That sounds familiar, unfortunately

felipefar• 2 years ago

Unfortunately it's not supported by Safari either.

venusenvy47• 2 years ago

I use SingleFile on Firefox quite often for this purpose. https://addons.mozilla.org/en-US/firefox/addon/single-file/

lukan• 2 years ago

I use it regulary. It works on static sites quite well, but subsites are not automaticaly saved, so not crawled.

unlog16• 2 years ago

Yes! You know, I was considering this the previous couple of days, was looking around on how to construct a `mhtml` file for serving all the files at the same time. Unrelated to this project, I had the use case of a client wanting to keep an offline version of one of my projects.

Although UNIX philosophy posits that it's good to have many small files, I like your idea for its contribution to reduceing clutter (imagine running 'tree' in both scenarios) and also avoiding running out of inodes in some file systems (maybe less of a problem nowadays in general, not sure as I haven't generated millions of tiny files recently).

Pretty rare for any website to have many files, as they optimize to have as few files as possible(less network requests, which could be slower than just shipping a big file). I have crawled react docs as a test, and it's a zip file of 147mb with 3.803 files (including external resources).

https://docs.solidjs.com/ is 12mb (including external resources) with 646 files

mikeqq20244• 2 years ago

trying to use this for mirroring a document site. disappointed at 1. it running quite slow, 2. it kept outputing error messages like "ProtocolError: Protocol error (Page.bringToFront): Not attached to an active page". not sure what reason

ajvs• 2 years ago

SingleFile extension is the modern equivalent these days.

marban• 2 years ago

I just opened an .mht file from 2000 on Edge/Mac the other day and it displayed just fine.

kitd• 2 years ago

Nice work!

Obligatory mention for RedBean, the server that you can package along with all assets (incl db, scripting and TLS support) into a single multi-platform binary.

https://redbean.dev/

ilrwbwrkhv• 2 years ago

Wow this is so cool. I like these types of single binary things. Even Pocketbase is like that where you can just compile your whole app into one Go binary and just rsync that over to your server and run it.

djbusby• 2 years ago

Go makes a binary for each target.

RedBean uses magic from @jart to run the same binary on multiple platforms.

tamimio2• 2 years ago

How is it different from HTTrack? And what about the media extension, which one is supported and which one isn’t? Sometimes when I download some sites with HTTrack, some files just get ignored because by default it looks only for default types, and you have to manually add them there.

unlog16• 2 years ago

Big fan of HTTrack! reminds me of the old days and makes me sad of the current state of the web.

I am not sure if HTTTrack progressed from fetching resources, long time since I used it for last time, but what my project does, is spin a real web-browser(chrome in headless mode which means it's hidden) and then it lets the JavaScript on that website execute, which means it will display/generate some fancy HTML that you can then save it as is into an index.html. It saves all kind of files, it doesn't care the extension or mime types of files, it tries to save them all.

tamimio2• 2 years ago

It saves all kind of files, it doesn't care the extension or mime types of files, it tries to save them all.

That’s awesome to know, I will give it a try. One website I remember I tried to download and has all sorts of animations with .riv extension and it didn’t work well with HTTrack, will try it with this soon, thanks for sharing it!

renegat0x02• 2 years ago

My 5 cents:

- status codes 200-299 are all OK

- status codes 300-399 are redirects, and also can be OK eventually

- 403 in my experience occurs quite often, where it is not an error, but suggestion that your user agent is not OK

- robots.txt should be scanned to check if any resource is prohibited, or if there are speed requirements. It is always better to be _nice_. I plan to add something like that and also missing it in my project

- It would be interesting to generate hash from app, and update only if hash is different?

unlog16• 2 years ago

Status codes, I am displaying the list because mostly on a JavaScript driven application you don't want other codes than 200 (besides media).

I thought about robots.txt but as this is a software that you are supposed to run against your own website I didn't consider it worthy. You have a point on speed requirements and prohibited resources (but is not like skipping over them will add any security).

I haven't put much time/effort into an update step. Currently, it resumes if the process exited via checkpoints(it saves current state every 250 URLs, if any is missing then it can continue, else it will be done)

Thanks, btw what's your project!? Share!

renegat0x02• 2 years ago

I agree with your points.

You might be interested in reddit webscraping thread https://www.reddit.com/r/webscraping/

My passion project is https://github.com/rumca-js/Django-link-archive

Currently I use only one thread for scraping, I do not require more. It gets the job done. Also I know too little to play more with python "celery" threads.

My project can be used for various things. Depends on needs. Recently I am playing with using it as a 'search engine'. I am scraping the Internet to find cool stuff. Results are in https://github.com/rumca-js/Internet-Places-Database. No all domains are interesting though.

PenguinCoder• 2 years ago

Status codes, I am displaying the list because mostly on a JavaScript driven application you don't want other codes than 200 (besides media).

What? Why? Regardless of the programming language used to generate content, the standard, well known HTTP status codes should be returned as expected . If your JS served site, gives me a 200 code when it should be a 404, you're wrong.

glandium• 2 years ago

In many cases, 403 is really 404 on things like S3.

meiraleal4• 2 years ago

Seems like a very useful tool to impersonate websites. Useful to scammers. Why would someone crawl their own website?

kej2• 2 years ago

Scammers don't need this to copy an existing website, and I could see plenty of legitimate uses. Maybe you're redoing the website but want to keep the previous site around somewhere, or you want an easy way to archive a site for future reference. Maybe you're tired of paying for some hosted CMS but you want to keep the content.

meiraleal4• 2 years ago

All the scenarios you described can be achieved by having access to the source code, assuming you own it.

unlog16• 2 years ago

Why would someone crawl their own website?

My main use case is that the docs site https://pota.quack.uy/ , Google cannot index it properly. On here https://www.google.com/search?q=site%3Apota.quack.uy you will see some tiles/descriptions won't match what the content of the page is about. As the full site is rendered client side, via JavaScript, I can just crawl myself and save the html output to actual files. Then, I can serve that content with nginx or any other web server without having to do the expensive thing of SSR via nodejs. Not to mention, that being able to do SSR with modern JavaScript frameworks is not trivial, and requires engineering time.

ibeckermayer• 2 years ago

I’m not quite understanding: you’re saying you deploy your site one way, then crawl it, then redeploy it via the zipfile you created? And why is SSR relevant to the discussion?

jbaber• 2 years ago

I want to take down a full copy of a site hosted on Squarespace before moving off of it.

I have no access to source and can't even republish the site directly without violating Squarespace's copyright.

But having the old site frozen in amber will be great for the redesign.

tomByrer• 2 years ago

I think you can also screenshot full length in Chrome-based browsers; do both desktop & mobile widths.

It would be a good backup for the backup, & you designer will thank you.

ivolimmen• 2 years ago

So a modern chm (Microsoft Compiled HTML help file)

nox101• 2 years ago

I'm curious about this vs a .har file

In Chrome Devtools, network tab, last icon that looks like an arrow pointing into a dish (Export har file)

I guess a .har file as ton more data though I used it to extract data from sites that either intensionally or unintentionally make it hard to get data. For example, signing up for an apartment the apartment management site used pdf.js and provided no way to save the PDF. So saved the .har file and extracted the PDF.

jraph• 2 years ago

IIUC HAR files contain an awful lot of data that you would not want to end up being stored in a web page archive:

- irrelevant http headers (including cache control) taking up too much space

- auth data / cookies, credentials, personal infos that you don't want saved for privacy and security reasons especially if you want to share your archive.

HAR is also not very efficient for this use case: it's a list of request represented in json. A folder representation is far better, for storage efficiency as well as for reading the archive (you basically need to implement some custom logic that rcan read the page replaying the requests).

ProtoAES256• 2 years ago

Wow! I never knew things like this existed! I always used wget (full below) but nowadays seemingly all sites are behind cloudflare so I need to pass a cookie too.

Glad to see easier methods!

  wget \
     --header "Cookie: <cf or other>"
     --user-agent="<UA>"
     --recursive \
     --level 5 \
     --no-clobber \
     --page-requisites \
     --adjust-extension \
     --span-hosts \
     --convert-links \
     --domains <example.com> \
     --no-parent \
         <example.com\sub>

szhabolcs• 2 years ago

How does it work with single page apps? If the data is loaded from the server, does it save the page contents as full, or just the source of the page?

unlog16• 2 years ago

It saves the generated/rendered html, but I have just added a `spa` mode, that will save the original HTML without modifications. This makes most simple web app work.

I have also updated the local server for fetching from origin missing resources. For example, a webapp may load some JS modules only when you click buttons or links, when that happens and the requested file is not on the zip, it will fetch it from origin and update the zip. So mostly you can back up an SPA by crawling it first and then using it for a bit for fetching the missing resources/modules.

zb3• 2 years ago

Note I've recently released a tool that can find all chunks of a SPA that works for popular configurations (like webpack or ES modules):

https://github.com/zb3/getfrontend

CGamesPlay• 2 years ago

I like the approach here! Saving to a simple zip file is elegant. I worked on a similar idea years ago [0], but made the mistake of building it as a frontend. In retrospect, I would make this crawl using a headless browser and serve it via a web application, like you're doing.

I would love to see better support for SPAs, where we can't just start from a sitemap. If you're interested in, you can check out some of the code from my old app for inspiration on how to crawl pages (it's Electron, so it will share a lot of interfaces with Puppeteer) [1].

[0] https://github.com/CGamesPlay/chronicler/tree/master [1] https://github.com/CGamesPlay/chronicler/blob/master/src/mai...

unlog16• 2 years ago

It tries to fetch a sitemap for in case there's some missing link. But it starts from the root and crawls internal links. There's a new mode added this morning for spa with the option `--spa` that will write the original HTML instead of the generated/rendered one. That way some apps _will_ work better.

billpg• 2 years ago

https://example.com/droste.zip/droste.zip/droste.zip/droste.......

sedawk• 2 years ago

I used to use MAFF (Mozilla Archive Format)[1] a lot back in the day. I was very upset when they ended the support[2].

I never dug deeper whether I can unzip and decode the packing, but saving as simple ZIP does somewhat guarantee future-proofing.

[1] https://en.wikipedia.org/wiki/Mozilla_Archive_Format

[2] https://support.mozilla.org/en-US/questions/1180271

Per_Bothner• 2 years ago

The libwebsockets server (https://libwebsockets.org) supports serving directly from zip archives. Furthermore, if a URL is mapped to a compressed archive member, and assuming the browser can accept gzip-compressed files (as most can), then the compressed data is copied from archive over http to the browser, without de-compressing or conversion by the server. The server does a little bit of header fiddling but otherwise sends the raw bytes to the browser, which automatically decompresses it.

Show HN: Crawl a modern website to a zip, serve the website from the zip

Related Stories