Show HN: Pdf-Bot, an API/CLI for Generating PDFs Using Headless Chrome

by esbenp • 9 years ago

235 79 9 years ago

vbezhenar• 9 years ago

Headless chrome is awesome. I'm using it to generate multiple PNG from SVG.

keyle• 9 years ago

I agree with you but as a 'old' developer, I find it pretty sad that you have to fire up a headless browser + its gigaton of code, to convert a SVG to PNG. But I agree with you, headless chrome can be very useful.

wibr• 9 years ago

You could also do that with inkscape on the commandline, not sure if it's faster, though...

bshimmin4• 9 years ago

I take your point in general, but I'm not sure this specific task - converting of SVGs - has ever been one that has easily been solved by some tiny amount of code: probably the way I would have done this in the past (say, ten years ago) would have been using Apache Batik's rasteriser, which is far from a lightweight solution itself.

_Codemonkeyism• 9 years ago

We currently use Apache Batik (JVM/Scala) to generate PNG for server generated SVGs of charts. SVG is wonderful for generating charts, easy even without any framework.

alfonsodev• 9 years ago

Indeed, I see lately many cool projects emerging. Is a pity that not all projects are using the same GitHub tag #headless-chromium [1]

This other project[2] was recently on HN but you can't find it easily on github search.

[1] https://github.com/search?q=topic%3Aheadless-chromium&type=R...

[2] https://github.com/DevExpress/testcafe

czechdeveloper• 9 years ago

All I need now is CMYK support in Chrome and I can make HTML based print ready PDF rendering. That would be quite upgrade compared to my current options.

hwc3• 9 years ago

I work on Chrome's PDF generation.

Please file a bug report if you think this is an important feature.

synthmeat• 9 years ago

I'll upvote both this and track bug report, if linked.

Yes, CMYK support is very tempting feature for the whole print industry.

umpox• 9 years ago

Pretty cool! Check out Google's headless browser API Puppeteer too, they provide a few really useful functions for doing stuff like this.

https://github.com/GoogleChrome/puppeteer/blob/master/exampl...

Really easy to work around, I used it to build a simple CLI to generate device screenshots of a webpage by modifying the user-agent and resolution to match each device.

https://github.com/umpox/generateDeviceScreenshots

esbenp4• 9 years ago

Yeah I think Puppeteer is a very cool project. Unfortunately, it came out literally one or two days after i finished the initial version of pdf-bot. Maybe I will incorporate it soon! :-)

rememberlenny• 9 years ago

Cool project

matallo• 9 years ago

idea: using this as "send to kindle" generating pdfs from urls you stumble upon, and seamlessly sending them to the *@kindle.com email address to consume in the device

I don't know if there's an easier way or service these days

rcarmo• 9 years ago

Instapaper did that pretty well, and I think Pocket Premium does that too - in MOBI format, which is much easier to read on the Kindle than PDF.

These days I use Calibre instead because the overall experience is better, but for single articles it's a bit overkill.

unmole• 9 years ago

I use the Push to Kindle app from FiveFilters.org to do this.

esbenp4• 9 years ago

That is a cool idea! Would only require to setup a small server as webhook endpoint that creates the e-mail.

pcnix• 9 years ago

I use a script that implements calibre's ebook-convert command line to convert and mail mobi files to myself from html urls. The ebook-convert command is powerful enough that I can join several html pages together, and also add chapter hooks and titles. I used to use this to download and read Wheel of Time rereads on Tor for a while.

bharani_m3• 9 years ago

Really nice work.

I've had a great experience working with Headless Chrome to convert webpages to PDF for my side project EmailThis (https://www.emailthis.me).

It uses Puppeteer by Chrome DevTools team - https://github.com/GoogleChrome/puppeteer.

esbenp4• 9 years ago

Appreciate it. EmailThis lookes very cool! I have considered using Puppeteer for pdf-bot, it came out right after I finished it :-)

RepressedEmu• 9 years ago

Just a heads up -- your site emailthis.me is down.

bharani_m3• 9 years ago

Sorry about the brief downtime. It's back up now.

johnwaynedoe• 9 years ago

Cool project! I think I am going to start regularly making use of it. Does it email you a PDF attachment, or does it just send an email with the contents of the article within it? When I attempted to use it I did not see a pdf attachment. Regardless/either way, really cool project.

bharani_m3• 9 years ago

Thanks, if it is unable to extract useful content from a page, EmailThis will save it as PDF as send it as an attachmentment.

If you try saving a any discussion website (HN, Stackoverflow, Reddit), you will get the PDF.

bm98• 9 years ago

Interesting to compare this to some of the "old school" solutions for converting web pages to PDF such as htmldoc[0] or html2ps[1].

[0] https://github.com/michaelrsweet/htmldoc [1] http://user.it.uu.se/~jan/html2ps.html

Wilya• 9 years ago

The old school solutions lack any sort of javascript support (per the docs, htmldoc doesn't even support css), so they wouldn't work for a lot of real world websites. That's not really the same use case.

A better comparison would be against the likes of wkhtmltopdf[0], which uses webkit, or the pdf generation features of phantomjs.

[0] https://wkhtmltopdf.org/

j_s5• 9 years ago

https://github.com/wkhtmltopdf/wkhtmltopdf/issues

1,047 Open 975 Closed

Yep, that's about how I remember it. It was such a pain to build on Windows (especially to get a single static binary) that people contributing fixes would often attain hero status by attaching a random binary to an issue. Specifically, GIF support was broken on the official Windows build for 4+ years:

https://web.archive.org/web/20140917181225/http://code.googl...

shubhamjain2• 9 years ago

I am guessing this works by splitting screenshots of a web page and gluing them as pages in the PDF file. Doesn't that mean the size of the PDF would grow to be large once it passes few pages? How does it handle content (like, tables) that don't have line breaks?

gima2• 9 years ago

It uses Chrome's built-in print-to-PDF functionality via Chrome Debug/DevTools Protocol. In other words it creates PDF files with real vector graphics and text, not just images embedded in PDF.

Page.printToPDF: https://chromedevtools.github.io/devtools-protocol/tot/Page/...

shubhamjain2• 9 years ago

I didn't know that existed. How good is it with corner cases? HTML->PDF is a notoriously difficult problem; even generating PDF is. There are several software services which charge well for doing that (Docraptor, PrinceXML). If it's smooth and handles everything well, is there any reason someone should pay for them?

criddell• 9 years ago

Can it generate a table of contents with page numbers?

aumerle• 9 years ago

calibre has been able to convert arbitrary HTML files to PDF with Table of Contents with page numbers, links, embedded fonts, arbitrary headers/footers for years, all rendered using WebKit, without a running X server, for years.

ebook-convert file.html file.pdf --pdf-add-toc

hwc3• 9 years ago

It can do what HTML can do. You can have a TOC with internal links. Those links should continue to work inside the PDF.

sdeframond• 9 years ago

Nice work !

Depending on your use case, I feel like you guys might be interested in http://weasyprint.org/. It is an open source HTML to PDF converter written in Python. It passes the Acid2 test and implements CSS Paged Media.

vegbrasil• 9 years ago

WeasyPrint seems awesome. Going to prototype with it later! Thanks

j_s5• 9 years ago

Always nice to discover another open source HTML rendering engine!

I recently discovered https://github.com/ArthurHub/HTML-Renderer formerly know as https://htmlrenderer.codeplex.com/

PDF generation [...] 100% managed (C#), High performance HTML Rendering library

igitur• 9 years ago

How does this compare to wkhtmltopdf, which, IIRC, uses WebKit to render the pdf?

esbenp4• 9 years ago

We actually used wkhtmltopdf before we started using pdf-bot. wkhtmltopdf development has slowed a lot, it is very unstable and you need to run a 2 year old alpha version to support flexbox (if I remember correctly) :-) headless chrome is a lot more stable choice imo.

bshimmin4• 9 years ago

We had an absolutely ghastly time last year trying to implement wkhtmltopdf in a Rails app - we probably wasted an entire week fighting with both Wicked PDF and PDFKit before we just gave up and wrote something using Prawn instead (which was, of course, extremely time-consuming in a different way, but at least the end result was good).

shimon_e• 9 years ago

Buggy as hell. Doesn't render things as expected. I'm using headless chrome for pdf generation internally and couldn't be happier. I directly call chrome from the command line it couldn't be simpler. I don't know why people need all these wrappers.

kyriakos4• 9 years ago

as far as I know wkhtmltopdf still exceeds in some aspects, better page control, margins, headers, footers and tables of contents. chrome still doesn't support them and also no browser supports CSS Paged Media which allows for setting up running elements like page numbering etc

flashman• 9 years ago

How well does it work with multiple-page PDFs? One of our banes is generating mixed text/image downloadable reports with sensible page breaks. To save time, we're actually doing those as docx files, with the bonus/risk that clients can edit the content before saving it as a PDF.

mnsc• 9 years ago

The holy grail of PDF generators, sensible page breaks. Never been done and makes peace in the middle east seem like an easy task.

cpr2• 9 years ago

PrinceXML does a solid job of it, and has for years.

DonnyV4• 9 years ago

Just use WkhtmlToPdf https://wkhtmltopdf.org and wrap a simple service around it.

bshimmin4• 9 years ago

That's one hell of a "just".

moocowtruck• 9 years ago

DonnyV don't you know we need 15 million of the same things in technology :)

kyriakos4• 9 years ago

wkhtmltopdf is a solid tool but its mostly abandoned now. rendering in chrome headless is way faster and more accurate. chrome headless though is lacking a lot of features that wkhtmltopdf provides like headers/footers

j_s5• 9 years ago

There's CSS that's supposed to help with doing this manually, right? Or does your docx export handle this auto-magically? (If so, I'm interested in more details!)

https://developer.mozilla.org/en-US/docs/Web/CSS/page-break-...

lucaspiller• 9 years ago

CSS isn't really that flexible though, say you want a repeated header or footer on each page, you are going to have a real hard time getting it to work perfectly. And good luck if you then need to internationalize it and support different document sizes (A4 vs legal). It's not impossible, but it's a lot more work than it should be.

One of the best ways I've used to generate PDFs is by using a DOCX as a template, and replacing certain placeholders within the document (a DOCX is a ZIP containing a few XML files). It's great if you work in a corporate environment, as it's easy for non-technical staff to make it look exactly how they want and it's easy to update (just replace a file and check it works). You can use headless LibreOffice to convert DOCX to PDF.

brajesh• 9 years ago

Are there any similar wrappers around headless Firefox, which has been released recently (Firefox 55)?

Mozilla's documentation (https://developer.mozilla.org/en-US/Firefox/Headless_mode) is still incomplete.

dagurp• 9 years ago

I assume they're waiting for Servo

rmetzler• 9 years ago

Thank you very much. I'm excited to try it out. Great documentation, I wish more people would explain their project's software architecture in the README.

titel• 9 years ago

Whould this work on AWS Lambda?

joshribakoff3• 9 years ago

Since it's designed to run daemonized as a queue I'd think you'd have an issue with execution time limit

benmanns• 9 years ago

Check out https://github.com/adieuadieu/serverless-chrome#printtopdf-p...

joshribakoff3• 9 years ago

What's up with the built in queue? I feel like that belongs in a different script. For one, the built in nodeJS queue is useless in a multiple server environment. You'd still need a distributed queue since this built in one is only local to one server/thread. So the built in queue becomes redundant/pointless for any kind of solution that needs to scale

chatmasta• 9 years ago

Not the OP, but maybe he wanted to keep the install as simple as possible without requiring something like redis. I also haven't looked at the code, but a queue per process is still useful as long as the results can be accessed from any process. Not sure if this is the case. If not, you would seem to be right, in that subsequent requests after "queuing" the job could go to another process/server not aware of the queued job.

xg15• 9 years ago

I was first wondering about all the complexity in the API as well (why a built-in queue, webhook, retry policy and storage interface when the actual transaction I'm interested in is just "url -> pdf blob"?)

However, I think this is necessary if you want to fit it into a microservice with a REST interface. For REST, I think the usual expectation is that a) the request returns quickly and b) you can submit any number of requests in parallel. Given that loading a page into headless chrome, rendering it and generating a pdf is both resource intensive and time consuming, I guess you need some way to decouple that process from the interface.

gbuk20132• 9 years ago

This is interesting :-)

I am currently using athenapdf[1] but I will have a play with pdf-bot.

[1] https://github.com/arachnys/athenapdf

MrSaints3• 9 years ago

Core developer of `athenapdf` here :)

I had a quick look at `pdf-bot`, and though we both rely on the same underlying technology (we are only just moving to headless Chromium; we were on Electron before), I believe we have slightly different ambitions with our respective project. But, I may be biased.

For example, `pdf-bot` seems to be tied exclusively to a specific converter, and storage backend. With `athenapdf` however, we are moving more, and more towards building a toolkit or rather, framework for other people to construct their own conversion processes (or even microservice)[0].

Consequently, we are working towards general abstractions like fetching, converting, and uploading, that can have different implementations (e.g. wkhtmltopdf, LibreOffice, Weasyprint, etc).

With our microservice assembly as well, we are focused heavily on ensuring we have:

1. Instrumentation, and metrics (which `pdf-bot` appears to currently lack)

2. Support for different retry mechanisms (e.g. retry using the same converter or retry using a different converter)

3. Support for multiple input MIME types

4. Synchronous API calls (`pdf-bot` appears to be mostly asynchronous, with batch processing, and callbacks)

5. Ease of installation (e.g. Docker), and configuration

We also have a CLI assembly[1] that can support custom JavaScript plugins[2] (e.g. Markdown -> PDF, Readability, etc). So you don't need to run a service or make API calls for conversions.

[0] https://github.com/arachnys/athenapdf/tree/v3/pkg

[1] https://github.com/arachnys/athenapdf/blob/v3/cmd/cli/main.g...

[2] https://github.com/arachnys/athenapdf/tree/v3/pkg/runner/plu...

gbuk20132• 9 years ago

Thank you for athenapdf and for rescuing me from the pains of wkhtmltopdf - I am a happy user. :)

My only small problem with it was the somewhat complex setup for using athenapdf-service with a new project (especially since I use docker-machine) but I have now mostly automated the whole thing.

Just out of interest - do you consider asynchronous an advantage (being a Node developer I generally love async very much)? Not that it matters to me - my needs are trivial for the service to handle.

Edit: actually I can see how it async would make my life much more complicated for my simple use case - I would have to write something to track requests and responses rather than just looping through a bunch of URL's that need converting.

kyriakos4• 9 years ago

hi and thanks for athenapdf

I was wondering, does it support custom page headers (or generally running elements)

oxplot• 9 years ago

Unfortunately, Chrome's kerning when it comes to printing is atrocious. Over the years, I've constantly tested it every once in a while with the hope that it would improve to no avail.

Currently, the only print ready HTML to PDF processor that I know is Prince [1] and to a lesser extent Firefox.

[1] https://www.princexml.com/

sk5t• 9 years ago

Prince is awesome. Its support for paged media has impressed me again and again.

kyriakos4• 9 years ago

does Firefox support css paged media?

Show HN: Pdf-Bot, an API/CLI for Generating PDFs Using Headless Chrome

Related Stories