Babashka is a fast-starting scripting environment for Clojure

https://medium.com/graalvm/babashka-how-graalvm-helped-create-a-fast-starting-scripting-environment-for-clojure-b0fcc38b0746

by tosh • 4 years ago

217 42 4 years ago

medium.com

timdeve2• 4 years ago

I've been using it for quick web scrapping scripts and it's really nice.

nerpderp822• 4 years ago

What libraries do you use? I do most of my scraping in Python using beautifulsoup.

aeonik• 4 years ago

Not OP but I use Reaver with good results. It supports all of JSoup's selectors, and makes it very clean to extract data from HTML.

The documentation is a little lacking though, I had to look up other examples on GitHub to figure out how to use all the features.

https://github.com/mischov/reaver

Borkdude• 4 years ago

Babashka doesn't have a built-in HTML parsing library but it supports it through pods:

https://github.com/babashka/pod-registry

Pods can be written in any language and they can expose functions to babashka by implementing a protocol.

One pod exposing HTML parsing is:

https://github.com/retrogradeorbit/bootleg

Here is an example of how to use that:

https://github.com/babashka/pod-registry/blob/master/example...

noblepayne• 4 years ago

As mentioned by the one and only Borkdude, bootleg is a nice option for this.

It includes the Hickory library: https://github.com/clj-commons/hickory

I'm a previous BeautifulSoup user and have found the combination of (1) having the scraped data presented in plain Clojure data structures, and (2) Hickory's built in selectors, to be a very nice experience.

Happy scraping!

nathell• 4 years ago

I plan to port my scraping framework (Skyscraper, https://github.com/nathell/skyscraper) to babashka one day. I’m not sure how easy it will be, though, since it uses core.async (which I believe bb has limited support for) and SQLite via clojure.java.jdbc.

timdeve2• 4 years ago

As other people have said Bootleg + Hickory.

Here is an, admitedly not very clean, example that grabs stream urls from hltv.org:

https://github.com/TimDeve/.dotfiles/blob/master/scripts/gen...

Also a basic RSS reader using the clojure XML lib:

https://github.com/TimDeve/.dotfiles/blob/master/scripts/gen...

nerpderp822• 4 years ago

Thanks everyone, all great pointers. I'll do my next scrape in Babashka!

bokchoi• 4 years ago

Babashka is great! I've used it for doing some munging of csv and xml files.

the-alchemist• 4 years ago

It's this really nice combination of bash, curl, and libraries for JSON, CSV, ZIP files, and almost everything else you could need. Full list at [0]. You can always spawn a shell too.

It starts just as fast as Python too, if not faster, at least on my machine:

   time python3 -c 'import os.path; print(os.path.exists("README.md"))'

   time bb -e '(.exists (new java.io.File "README.md"))'

Python averages around 40-50ms, and bb averages 30-40ms.

[0] https://book.babashka.org/#built-in-namespaces

capableweb3• 4 years ago

Python averages around 40-50ms, and bb averages 30-40ms.

Sounds like you have a slow harddrive here, rather than measuring the startup of the interpreters.

With hyperfine:

    $ hyperfine --warmup=100 --runs=1000 'python3 -c "import os.path; print(os.path.exists('"'README.md'"'))"' 'bb -e '"'(.exists (new java.io.File "'"README.md"'"))'"''
    
    Benchmark 1: python3 -c "import os.path; print(os.path.exists('README.md'))"
      Time (mean ± σ):       5.3 ms ±   0.7 ms    [User: 4.9 ms, System: 1.2 ms]
      Range (min … max):     4.5 ms …   9.8 ms    1000 runs
    
    Benchmark 2: bb -e '(.exists (new java.io.File "README.md"))'
      Time (mean ± σ):       6.0 ms ±   2.5 ms    [User: 2.5 ms, System: 4.3 ms]
      Range (min … max):     0.4 ms …  10.6 ms    1000 runs
    
    Summary
      'python3 -c "import os.path; print(os.path.exists('README.md'))"' ran
        1.12 ± 0.49 times faster than 'bb -e '(.exists (new java.io.File "README.md"))''

Running on the following "not absolutely shabby" specs:

Intel i7-1185G7, 6.0.11-arch, Toshiba NVMe SSD (KBG40ZNS256G)

Seems Python has more consistent startup performance (`4.5 ms … 9.8 ms`) while Babashka can startup faster (`0.4 ms … 10.6 ms`) but on average, Python startups faster.

__jem• 4 years ago

bb is 1.32 ± 0.08 times faster on my computer (m1 mbp).

charesjrdan• 4 years ago

I think this is one of my favourite pieces of tech in the past five years tbh.

I still use bash for short <5 line scripts but everything else is bb (though I’ve started looking into nbb because you can use node libs like ink which seems pretty cool)

And repl integration with neovim and conjure is great!

musha68k• 4 years ago

I personally stopped using it since both shellcheck and jq fill their own niches way too nicely “unfortunately”…

None the less, a lovely tool and the couple of scripts I wrote are mostly still in use and well maintained for a reason ;) Clojure is just fantastic and so is the speed of the GraalVM.

pixelmonkey• 4 years ago

This is a particularly joyful piece of open source software. After seeing Michiel Borkent -- its author and writer of OP -- present on it at a meetup, I wrote a tidy overview of babashka (bb) which you can find here:

https://amontalenti.com/2020/07/11/babashka

kopos• 4 years ago

All our cron jobs and scripts with non trivial logic are in babashka now. It has been a joyful experiences

__MatrixMan__• 4 years ago

The positivity in the comments here makes me want to learn Clojure. Apparently it makes you happy.

capableweb3• 4 years ago

It absolutely makes you happy. Be beware, finding a language that makes you too happy, can lead to you not wanting to program in anything else anymore. You're better off being in the middle of grumpy and happy, like a normal programmer.

eyelidlessness• 4 years ago

Don’t worry, folks! Clojure will make you happy, but even with fast perf you may find yourself wanting static types or even just dynamic runtime result/option types. Of course it’s a lisp with macros, and a general purpose language, so you can have all of that too. But there’s no shortage of green in other pastures ;)

fud1012• 4 years ago

this is why i avoid common lisp now. for years i hated programming until i finally decided CL was dead to me. Now i happily write Python for a living and don't care it sucks relative to CL.

lordgroff• 4 years ago

Don't learn a Lisp. For one, you'll end up in endless arguments whether your Lisp is a proper Lisp, and two, everything else will look ludicrously constraining for no good reason.

yladiz2• 4 years ago

Lisp-1s are bastard children of the true Lisp-2s /s

fud1012• 4 years ago

this is good advice. its a trap. you live in a world where lisp isn't going to be accepted, for better or worse, lisp isn't going to happen.

sokoloff2• 4 years ago

I'm doing Advent of Code in clojure for the second year. It's frustrating in some ways (learning the string-parsing to process the inputs takes longer to learn in clojure than to simply execute in C++, C#, or JS, but that's because I know those other languages much better).

But the pleasure of iterating towards the solution and building and quickly testing each of the constituents is joy-inducing.

wry_discontent• 4 years ago

That's funny because I've found the exact opposite. Most everything seems to use roughly the same utilities, but Clojure lets me move more quickly in getting the inputs parsed correctly.

For example, I often start with something like `(slurp "inputs/day-2")` and keep wrapping that until I get a structure I'm happy with. Then I split it into a few fns so I can keep testing with the smaller input

eamonnsullivan• 4 years ago

I used this for my back up system: https://github.com/eamonnsullivan/backup-scripts

The server runs on a Raspberry Pi with a 1-2TB USB disk attached.

user3939382• 4 years ago

I've been looking for a good excuse to use Clojure, small shell scripts sounds like a great low-risk way to do that.

lukaszkorecki• 4 years ago

BB is awesome - it powers our RabbitMQ management scripts, we have written a sidecar process to scrape metrics from ECS and send them to vector.dev over UDP and a bunch of other things.

For personal stuff I use it in a lot of scripts used by Alfred.app including quick-fill for Things with date parsing, emoji completion, etc

whacked_new• 4 years ago

babashka is so good it's mind-boggling.

super-easy shell interop + instant nREPL = replaces any complex bash script, driven from emacs!

what's more, you can namespace your scripts and import them. functionalizing code makes scripts easy to refactor, or even trivially move into a full Clojure backend or ClojureScript frontend. With squint-cljs you can even generate self-contained JavaScript directly from babashka code.

ducktective2• 4 years ago

Is there something similar for Common Lisp?

comma_at• 4 years ago

I think SBCL starts up fast enough as-is, no? If you want a batteries included version with libraries, just load them and save a new base image with SAVE-LISP-AND-DIE

vindarel4• 4 years ago

No, we have to build a binary, which starts up super quickly.

I began to put together a "distribution" of useful CL libraries for everyday tasks: https://github.com/ciel-lang/CIEL/ It comes as:

- a lisp core, which you can use in your editor setup instead of sbcl or ccl, the advantage is that it loads instantly with all these libraries built-in (instead of quickloading all of them when needed)

- a readline REPL, with completion, syntax highlighting and some goodies (a shell passthrough, quick doc lookup…), albeit it needs more love

- a lisp library which you can quickload like the rest, it just comes with a bunch of dependencies.

- I want to add scripting capacity too.

It ships libraries for: HTTP requests, file formats handling (JSON, XML, CSV, TOML…), shell utils, file utils, threads, a nice hash-table notation, DB access, plotting, strings, other language extensions etc.

It is not finished but I am doog-fooding it for clients' projects.

ducktective2• 4 years ago

Thank you very much!

So the scope of it is like Janet-lang, right?

vindarel4• 4 years ago

BTW, Roswell makes it easier to run scripts: https://github.com/roswell/roswell/wiki/Roswell-as-a-Scripti...

It is also a tool to install various CL implementations, and to install software.

It doesn't come with a choice of built-in libraries.

Babashka is a fast-starting scripting environment for Clojure

Related Stories