Import – A Simple and Fast Module System for Bash and Other Unix Shells

by pcr910303 • 6 years ago

129 50 6 years ago

meddlepal• 6 years ago

Never change below your feet unless your machines are ephemeral, constantly changing, and thus do not have a permanent cache.

Do not use this in production folks.

SrslyJosh• 6 years ago

The bare minimum for saying "never change" involves a cryptographically secure checksum of the content, and even then it's not guaranteed.

price3• 6 years ago

Exciting idea!

For me the application of this that I'm most interested in personally is for dotfiles, configuring the interactive shell. I have a fair amount of useful stuff in my own bashrc that I've developed over the years, which people who see me work often want to use too (and I happily share). I've wished for a good way to share each logically-independent customization in its own separate chunk -- so that people can pick and choose, and also combine what they like from me with their own existing config and chunks they've picked up elsewhere.

For zsh, oh-my-zsh seems to have been extremely successful, and IIUC this describes a key technical contribution at the core of how it's worked so well.

So I'm hoping that this thing can play part of the role of oh-my-zsh, for bash! (And for other shells too, sure.)

I'll definitely be interested to read more of the details of how it works, and to play around with it.

price3• 6 years ago

(Before using this for real, I'd certainly want to be able to track the exact version I'm getting of a dependency, with a hash, in a "lockfile" sort of file that I can keep in version control. That's not mentioned in the demo on the front page - but if it doesn't already have that feature, I hope it will grow it.)

JoshTriplett2• 6 years ago

If you're pinning a shell script to a specific hash, why not just keep the script itself in version control?

boogies2• 6 years ago

As https://news.ycombinator.com/item?id=23865244 mentioned, the link should be able to point to anything, including a specific commit in version control.

chriswarbo• 6 years ago

As mentioned elsewhere, Dhall has a similar import-from-URL feature (with cryptographic hashes), and it is/was popular to use IPFS URLs, since they're content-addressed (i.e. changing the content will change the address, so each address will always point to the same data)

j1elo• 6 years ago

You might be able to make your own "chunk modules" system, by sourcing from .profile something like ".shell_modules//.sh", and then use GNU Stow [0] to selectively install, from a centralized directory, whatever modules you're interested in.

Each module would have to go to its own subdir (the '*' part in my example), but that's a good thing as it means that each module could contain several files.

[0] https://www.gnu.org/software/stow/

JoshTriplett2• 6 years ago

Seems to be failing at the moment: https://imgur.com/a/UEbJBKA

boogies2• 6 years ago

It was the same for me the first time I tried, but the "View on GitHub" link at the bottom still worked (and now the main page works too).

kazinator5• 6 years ago

   # The URL is downloaded once, cached forever, and then sourced
   import "https://git.io/fAWiz"

Where is this program declaring the above content's expected SHA256 to check that the expected content is being received, and not some altered version?

Think about it. You write the program with the above #import, and at the time, it fetches a certain piece of code. You see in your cache that it's the right one.

Then you deploy the script to a fellow user. They run it, and the import fetches something different from the same URL. The script breaks. You're scratching your head: hey, it works on my machine, why not yours? Or worse: the replacement code is malicious not simply breaking. The user's files are gone or whatever.

Spivak2• 6 years ago

Don't get me wrong, that's would be a nice feature, but I think it's a little optimistic that people would actually use it and ask for more security outside of TLS. I mean the most popular (n)vim plug-in managers just clone arbitrary git repos with no other verification.

Code signing or hashing is vanishingly rare and a huge burden on the end user who simply, unfortunately, not care in the slightest. The "good enough" root of trust is that the server hosting the code hasn't been compromised.

jiggunjer• 6 years ago

It's not so much about security as about reproducibility. What if this is being used in scientific research?

kazinator5• 6 years ago

Far from this being rare, every major GNU/Linux distro checks the hashes of downloaded material it depends on. Failing to do this threatens integrity/stability of the system and provides an open door for malware.

Vim package managers pulling git repos can easily check out a specific hash; if they are just blindly following branch heads by name, that's just dumb and nobody should emulate their behavior.

TLS is neither here nor there because there isn't much of a privacy concern. Plaintext HTTP works fine for fetching some piece of FOSS, if you check the digest to verify that you're getting what you're expecting to be getting.

On the contrary, the assurance that the HTTPS host key has not changed is not relevant. It does not amount to (is something other than) the relevant assurance that the URL content has not changed.

Moreover, there may be numerous hosts which provide that exact same content. If you have the digest, and the original host stops providing the content reliably, you can confidently switch to a different mirror, which could be a HTTPS host you have never contacted, or even a HTTP one.

The trust that the server hosting the code has not been compromised is misplaced and foolish. The person in charge of that content could be someone who thinks it is a good idea to change the content of a software release (to sneak in one more bugfix) without changing anything in the URL. Thus two of your own users who think they are using the exact same version of your software will fetch the same foolib-1.2.tar.gz URL a day apart, and get different files, which breaks the integrity of your own release. Without hashes, this goes undetected.

There is no "huge burden" here. All the module tool has to do is to make it mandatory for the fetching commmand to have a digest argument. If the digest doesn't match the content, all it has to do is fail that command, and issue a diagnostic. The diagnostic can include the actual hash that would work, and point to the file and line number where the failing import is located.

The user relies on hundreds of imports that they didn't write themselves, and which require no attention at all in the happy case that they fetch what they are supposed to fetch.

vmchale• 6 years ago

Dhall does this for import caching.

ary2• 6 years ago

  # The URL is downloaded once, cached forever, and then sourced
  import "https://git.io/fAWiz"

Call me old and cranky, but I thought this naive fad of piping the unchecked contents of URLs to the shell would die off eventually once people got clued in to how dangerous it is. At a bare minimum the contents of any shell script coming from a remote source should be manually reviewed the first time, and then verified to have not changed by comparing against a locally stored cryptographic hash.

This is no better than curl with some automatic caching. At the end of the day why not store your dependencies along side your scripts and then not worry about it changing unless you change it yourself?

endgame• 6 years ago

The dhall language does https imports, but with two important differences:

1. The language is _pure_ and _total_, so it will _eventually_ terminate and cannot perform side-effects 2. You can specify the SHA256 of the referenced resources inline, next to the URL. This gets you caching and breakage warnings.

treve2• 6 years ago

Piping a url into bash is not more or less risky than downloading a windows binary, installing a snap, installing a npm/composer/whatever package with post-install hooks, downloading a binary for your Mac.

It's impossible to read every line of code we execute (many of which are closed source anyway).

Downloading a large script and reading it before running it in bash is also hardly good security measure.

It's much more practical to teach people how to recognize trustworthy sources, and how HTTPS works. Or you can switch to a platform like iOS where there's complete control, sandboxes and reviews.

ary2• 6 years ago

The risks involved with executing code are legion, and wanting to promote cultural change in the direction of better safety isn't invalidated by pointing out failings in other areas.

Downloading a large script and reading it before running it in bash is also hardly good security measure.
It's much more practical to teach people how to recognize trustworthy sources, and how HTTPS works. Or you can switch to a platform like iOS where there's complete control, sandboxes and reviews.

Downloading bash scripts and inspecting them before running is definitely a good security measure, as is running untrusted software in a VM or container where possible, and generally promoting caution when running anything acquired from the Internet. Even if the source is considered trustworthy there is still the possibility that it has been compromised. npm, pypi, and even the Linux kernel source tree are all examples of this.

In this particular situation the use case is extremely weak. The problem is solved by incorporating dependencies locally, as I pointed out above.

radarsat1• 6 years ago

Still, downloading and executing code can be guarded by trusted signatures, as used by deb etc. You can protect such an "import" statement by restricting it to signed files, at the very least. That allows you to "trust" a source without reading every line of it. Not infallible, but much better than "download and run this thing from some obfuscated URL and assume it hasn't been hijacked"

choeger3• 6 years ago

Piping a url into bash is not more or less risky than downloading a windows binary, installing a snap, installing a npm/composer/whatever package with post-install hooks, downloading a binary for your Mac.

Wrong. Even windows binaries are signed nowadays. When you use a package manager, you cannot be targeted (assuming the package manager uses https) so you get at least some safety from sheer numbers. A URL, otoh, can easily be personalized. I could easily serve you a specific malware whereas everyone else and GitHub just sees the benign code.

easterncalculus• 6 years ago

Downloading a large script and reading it before running it in bash is also hardly good security measure.

I don't know how large you mean, but things written in the shell are usually very conducive to a quick scan. You're operating at a high level of abstraction so underhanded code is forced to be more obvious. It's actually extremely easy to find code that is at least suspicious enough not to run (but not necessarily malicious) so there's plenty of reason to still do it as opposed to trusting leaps of faith.

I guess if it's several thousand lines then it's not worth reading, but what script that's worth running is?

The other thing this misses is that you're running unsigned software, which despite some protesting I've seen here is certainly better than blind trust. That's what you're getting out of most binaries on your system, and other packages being pulled down. Piping a url into bash is more risky than running something unsigned, always.

HTTPS is also a pretty poor integrity enforcement tool. Sites are breached all the time making it trivial to change the code being served. You might not have anyone changing it on the way, but if you're drinking bad water straight out of the tap it doesn't matter.

earthboundkid• 6 years ago

Your scientists were so preoccupied with whether or not they could &c

edw• 6 years ago

Color me old, but wtf, the JS script kiddie obsession of never writing a line of code that can be packaged in a module has now come to shell scripting?!

choeger3• 6 years ago

Hello old, nice to meet you. I am skeptical.

aeyes• 6 years ago

Nice!

Who says that you have to download from arbitrary URLs? If you have to use public modules you are free to copy the source to some internal server first. It's just a single file!

It looks like modules can't import other modules so this shouldn't get too out of hand.

coredog64• 6 years ago

At my last employer (large US bank you’ve definitely heard of), they had something like this internally. Tools/scripts were packaged up as modules and you could/would “module load foo”. It knew what OS you were running, so Windows users could do the same thing for cross-platform tools.

It was very handy and I’m looking forward to trying this as a substitute.

alganet• 6 years ago

I did a similar thingy a while back:

https://github.com/alganet/coral/blob/master/doc/spec/requir...

It also does references (`require `something from inside modules) and some stuff. It is missing the download component though.

I've also worked on https://github.com/Mosai/workshop, a previous attempt at shell tooling (please do not use it!)

Also, there was a popular https://github.com/pixelomer/bashpm at some point.

Most stable thing would probably be: https://github.com/shellfire-dev

Here are my thoughts working in this "hobby" for the last 6 years:

- External tools are bad for shell portability and performance. It costs more to invoke "cut" than to do a loop and a couple of parameter expansions. Tools like cut, sed and similar are not suited for small strings, they are for large files.

- ksh93 and ksh2020 support is possible, see: https://gist.github.com/bk2204/9dd24df0e4499a02a300578ebdca4... These kshs are fast, I want them as supported engines.

- Pipes are bad for performance. Local variable buffer is faster in all shells. There is a proof here somewhere: https://github.com/alganet/shell-proto

- We need to create a shell "dialect". Leave a lot of stuff out to simplify its syntax, so we can write a shell parser written in pure shell. This is a crucial point towards a better shell ecosystem.

I have some research related to these issues, I'd love to collaborate in something. Standards, maybe? I think the shell community needs to talk as much as the early JS community did in the past.

varbhat• 6 years ago

I think that this is bad idea and it increases non-working scripts(what if url stops working? ) . Also, it increases malwares.(what if url pointed to is changed to malware script?)

Golang's package management is very good, but this isn't.

Why can't they just paste the required script in main script itself? What's the necessity of import here?

maddyboo• 6 years ago

This would be perfect if it also included a bundler/linker to build single-file executables.

I also find it amusing each time the argument against using URLs as import specs comes up. Go already does this, we do it on the web with <script> tags, and now Deno is doing it. It's proven to work as long as you can reasonably trust the URL.

idclip• 6 years ago

this makes me uncomfortable...

reasons:

1. i dont believe package systems help bash. we already have ansible for systems ... if this goes the way of npm, bash will get bloat and chaos.

2. shells are a system process thats usually very privileged .. i see alot of abuse potential since it interprets everything you source.

3. see 1, and think performance on medium sized scripts, or if an expensive "module" is loaded.

4. versioning chaos, killing old scripts ... in the unix and linux space, some decade olds scripts are still used.

this would expose weaknesses ... but also destabilise a lot.

edit: 3,4, also thank you regardless of my opinion, your code and execution look and feels great.

lma21• 6 years ago

what's wrong with

    source <file_path>

kasabali• 6 years ago

Not webscale /s

risson• 6 years ago

I hate the way a lot of tools nowadays assume it's okay to just create a new directory in the user's home. For instance, this tools create ~/.import-cache. Why doesn't it use a subdirectory in ~/.cache? What's the next step? ~/.import-config? And then ~/import-tmp? This is just getting out of hand. There are places where your tool can store config and cache data, use them.

augustk• 6 years ago

Or even better: $XDG_CACHE_HOME

https://specifications.freedesktop.org/basedir-spec/basedir-...

remix2000• 6 years ago

Impressive…

How people manage to poison even old-school shell scripts with useless bloat.

If you want to source remote scripts that badly (despite this being against policies of almost every company), better keep it simple:

    telesource() { f=~/.cache/${1/\//} ; [ -f "$f" ] || curl -SsLo "$f" "$1" && source "$f" ; }

So you can invoke it like that:

    # The URL is downloaded once, cached forever, and then sourced
    telesource https://git.io/fAWiz

What does the name ``import`` even mean? Does it anyhow relate to sourcing? Or even shell scripting at all?

Now I'm waiting for a full JIT ECMAScript VM in pure portable Bash. Impress me even more ;)

Nonetheless, that's what it looks like to me. I'm just not fond of spilling idioms from one technology to everything else its users touch. That's much like our IT equivalent of Midas touch.

emiliovesprini• 6 years ago

What does the name ``import`` even mean?

import name-clashes with ImageMagick's import, which comes pre installed in lots of Linux distros.

Issue (by me): https://github.com/importpw/import/issues/35

Import – A Simple and Fast Module System for Bash and Other Unix Shells

Related Stories