Entr: Rerun your build when files change

https://jvns.ca/blog/2020/06/28/entr/

by zdw • 6 years ago

202 81 6 years ago

jvns.ca

esperent• 6 years ago

This tool seems like a slight improvement on inotify.

Otherwise, it doesn't seem especially unusual or unique to me, and yet the author is presenting it as a revolution.

Am I missing something?

gridlockd• 6 years ago

A slight improvement in usability can make all the difference in the world for a user's experience.

cespare• 6 years ago

You're missing the fact that another person's knowledge or experiences could be different from your own.

mnw21cam• 6 years ago

Yeah, it's https://xkcd.com/1053/ happening yet again.

0mp2• 6 years ago

entr is cross-platform and utilizes the best of the platforms it runs on.

Also, the docs are explaining the problems with inotify: http://eradman.com/entrproject/

inshadows10• 6 years ago

You're not. It's glorified without any substance.

nickcw• 6 years ago

Nice tool - I'll be putting that in my toolkit.

I love the way she straces the tool to find out how it works - a true hacker at work :-)

wheresvic4• 6 years ago

Entr is amazing - I'm using it in my go projects to hot reload when developing!

https://smalldata.tech/blog/2019/03/15/hot-reload-go-applica...

rkwz• 6 years ago

Cargo Watch is something similar in the Rust ecosystem - https://github.com/passcod/cargo-watch

stefanchrobot• 6 years ago

Similar tool that I currently use: https://github.com/watchexec/watchexec

mattgreenrocks• 6 years ago

Original author here, glad you like it!

watchexec was borne out of a few frustrations with entr, mostly around how it handles new files being created. However, from a pure design standpoint, entr is just better than most anything out there due how closely it hews to UNIX philosophy, and it gets so far on just that.

The only real improvement that can be made to these tools (currently) would be to have perfect information about what caused a file to change. Currently, most tools require you to tell it files/patterns to ignore to avoid triggering loops where the file watcher changes files and ends up triggering itself over and over. watchexec did good work here by ingesting your .gitignore and using that.

Unfortunately OSes don't provide great info when it comes to file modifications. On Linux, ptrace/LD_PRELOAD would enable us to know the set of all files changed as a result of running the file watcher (and thus ignoring them automatically). DYLD_INSERT_LIBRARIES is a thing on macOS, though it is subject to SIP restrictions with some binaries. I'm unsure what mechanism exists on Windows. The highly platform-dependent nature of this is one reason why I haven't really pursued this line of work in watchexec.

psxuaw2• 6 years ago

Thank you for watchexec. It makes my life better every day.

jez2• 6 years ago

This seems to fill a similar role as watchman[1] (specifically with watchman-make[2])

I've heard good things about watchman because it will make a best effort to let filesystem changes "settle" before running the command specified. If there's a comparison of these two written up or if someone can give their testimonial I'd love to hear it.

[1] https://facebook.github.io/watchman/

[2] https://facebook.github.io/watchman/docs/watchman-make

michael-ax• 6 years ago

dependency free, solid design, perfect with tmux, great online reference. does not try to do magic: http://eradman.com/entrproject/

masklinn• 6 years ago

You do realise that's the tool TFA is about right?

jez2• 6 years ago

This post got me thinking that it'd be interesting if instead of being given an explicit list of files to watch for changes to, the list of files were inferred with an LD_PRELOAD that listened for `open` / `stat` / etc. system calls that a process ran.

For example, in the example on the blog post, `git ls-files` will almost certainly ignore autogenerated build files, but it's possible for one of those files to change without the output of `git ls-files` changing. Similarly for things like third party packages that are installed system-wide.

With an LD_PRELOAD, all you'd need to do is

    my-watcher ruby foo.rb

and the watcher would figure out which other Ruby files were opened, be them git versioned Ruby files in the current folder, or Ruby gems / Ruby VM dependencies in $HOME/.rbenv/, or system packages in /usr/local, or config files in /etc, ...

I guess I wouldn't actually be surprised to hear if someone has built this already.

thechao3• 6 years ago

fabricate.py (https://github.com/brushtechnology/fabricate), based off the now ancient memoize.py.

In my experience with C/++, it is faster to combine Make & ccache: just have every C file depend on every header file, and let ccache decide if it needs to be rebuilt.

AshamedCaptain2• 6 years ago

I really disagree there, specially as the project gets complex enough. ccache will not speedup anything related to linking, for example. But it will increase mtime of all your object files as it writes them even if they come from cache. So at some point even archive file creation dominates your "incremental" build time.

I suggest you don't wait until it's too complex to fix the mistake and do proper dependency tracking since the very beginning. It's not that hard in C/C++.

eatonphil• 6 years ago

Cool idea! Minor tweak though I'd rather have the watcher run separately, but I think that would still be easy enough if you resolved file paths to the project root.

cecilpl2• 6 years ago

I built a dependency checker that worked similarly to this once.

By hooking the filesystem calls, you can make a list of all files that a given process touches. When that process finishes, serialize a dependency file containing the hashes, timestamps, and sizes of all those files. Next time you run that same command line, read the dependency file from the last run and compare to the current filesystem state. If it's the same, and you know your command is idempotent, you can skip execution entirely.

Now, if you put that logic in a dll, you can inject it into arbitrary third-party processes to which you don't have source code and it will still work. Name the dependency file after the hash of the command line.

phone8675309• 6 years ago

The ClearCase SCM does something similar - it has the notion of "derived objects" and "audited builds".

You can run an audited build command under a special ClearCase wrapper and it will look at the versions of the elements used in the build - if that build has already occurred with the same input elements before, even if in a different view, it can "wink in" the derived object result - that is, it can cause that previous build to be visible in your current view. This can save a lot of time when you're building a large codebase.

AdieuToLogic• 6 years ago

A less intrusive approach could use kernel queues[0] on most Unix-like systems. This way, LD_PRELOAD would not need to be used and the process for responding to desired disk I/O is independent of the processes performing same.

0 - https://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&s...

MrBuddyCasino• 6 years ago

Some build systems work that way, tup is one I think. They use strace to intercept file I/O and figure out what has been updated, and thus can figure out an optimal way to re-build.

aidenn0• 6 years ago

I think Tup uses FUSE rather than strace (which is why it doesn't track external dependencies, and requires relative paths for all internal files), but I might be wrong.

RMPR• 6 years ago

Very good piece of software

Tip: Emulate vscode Live preview of Markdown with entr:

    ls file.md | entr script.sh

script.sh

    #!/bin/sh
    pandoc file.md -o file.pdf
    mupdf file.pdf

0mp2• 6 years ago

entr is a great too, I use it all the time to automate various little things when I'm developing. The coolest part that it works in every environment I care about, which is FreeBSD, Linux, and macOS.

imurray• 6 years ago

Commands like this are super-useful, and I use them all the time when editing LaTeX papers.

I used atchange for a long time:

http://jeffreycopeland.com/work/PDF/1996-03-atchange.pdf http://users.fred.net/tds/lab/atchange.html

And now, out of habit, my own wrapper around inotify. But more polished solutions like entr do seem the place to start now.

psxuaw2• 6 years ago

Another great tool that I use everyday is watchexec - https://github.com/watchexec/watchexec.

DelightOne• 6 years ago

https://github.com/cespare/reflex works well too.

globular-toast• 6 years ago

I use this kind of thing regularly with web stuff like Django, Flask and frontend/JS stuff using parcel, but I wonder what other things this is appropriate for? I can't imagine it being very useful for C/C++ code. I don't always want to rebuild every time I save a file.

emmelaich2• 6 years ago

The inotify cron daemon (incrond) is a daemon which monitors filesystem events and executes commands defined in system and user tables. It's use is generally similar to cron(8).

dmux• 6 years ago

Thanks for sharing this. I'm continuously surprised by how many tools exist in the *nix ecosystem that I've never come across. It makes me question if there's a discoverability problem here. Is there any website that broadly categorizes what utilities are available given a desired goal?

emmelaich2• 6 years ago

I've been meaning to 1. do a blog for the 'command of the day' and also 2. categorising commands by use / intent.

But until then, no I don't know any.

amelius• 6 years ago

There are better tools which use the ptrace api to check if files changed which the build actually depends on.

One downside is that ptrace is not recursive/reentrant so some tools won't work.

inshadows10• 6 years ago

Entr and many other similar alternatives do not satisfy my needs. The main fault is that they take list of files instead of root directory and possibly include/exclude patterns. So when I create or rename files, these tools either don't register new files, or they just fail on missing files.

I'm rather reusing existing stuff but seeing how every scriptable filesystem watcher misses the point, I'm inclined to write my own inotifywait/inotifywatch wrapper.

pull_my_finger• 6 years ago

Have you tried Tup[1]?

[1] http://gittup.org/tup/

codefined• 6 years ago

Nodemon[1] is also good, normally Node centric but all parameters can be configured.

[1] https://nodemon.io/

akx• 6 years ago

That's literally in the article; `entr -d`. The manpage also has an entry. http://eradman.com/entrproject/entr.1.html#d

inshadows10• 6 years ago

Terminal 1:

    mkdir /tmp/project
    cd /tmp/project
    touch file{1,2}
    ls | entr -d echo "a change"

Terminal 2:

    rm file2

Terminal 1:

    entr: cannot open 'file2': No child processes

Regarding `-d`:

Track the directories of regular files provided as input and exit if a new file is added. This option also enables directories to be specified explicitly. Files with names beginning with ‘.’ are ignored.

So first, it doesn't track NEW directories, and second, it exits if new file is added. Exactly how is this useful?

EDIT: All I want from filesystem watcher is to track files by pattern and just re-run the command, however complex that may be to implement via OS interfaces. When I'm working on a Python project that has packages (directories), I may also refactor (rename files), and this simplistic behavior does not catch that.

hnlmorg3• 6 years ago

I've ran into this issue myself and it's not filesystem watchers missing the point, it's actually something not supported in a number of operating systems. Thus you then either have to resort to nasty user-space hacks (which will be comparatively slow and resource hungry) or accept that you can't watch nested directories.

A more technical description is mentioned here: https://github.com/fsnotify/fsnotify/issues/18

inshadows10• 6 years ago

inotifywait can do this just fine. Here's an emulation of `entr` with an `include pattern`:

    inotifywait -rq -m -e close_write --format %f . | grep '\.rules$'

Sure, there may races due to limitations of underlying OS interfaces, but no such races really matter when you edit with mammal speed, so it's perfectly sufficient, and so it is perfectly possible to make good file system watcher, without any user-space power-hungry hacks.

pnathan• 6 years ago

here you go: it looks at everything in a specific directory.

    `while inotifywait -q -q -r -e modify -e move -e create -e delete --exclude '[*~#]' $(mapi $wd $@); do`

wd is the working directory.

and mapi

    my $prefix=shift;
    foreach my $s (@ARGV) {
           print "$prefix/$s ";
    }

Adjust to taste.

it is _hax_, as it were. but it has worked well for me.

seemslegit• 6 years ago

This should go nicely with that new js bundler and that new package manager.

je42• 6 years ago

Mmh. interesting. I have been using watch + redo to do this.

ivanstojic• 6 years ago

Entr requires one watch per file. One tool that I haven't seen mentioned here yet is fswatch, which uses directory based patterns and works more akin to a find: it outputs changed file names to stdout.

Interesting thing to look at and see if it maybe suits you better.

q3k• 6 years ago

If you're using Bazel, bazel-watcher [1] takes care of this, and does it in a much smarter way than just looking at a predetermined list of files.

[1] https://github.com/bazelbuild/bazel-watcher

rsanheim• 6 years ago

Guard [1] is a great tool in the ruby world for rerunning tests, reloading your browser, or running any arbitrary task based on files changing inside your file system. Its built on top of Listen [2] to do the fs piece, which has pretty well supported at least across linux and mac os over the years.

[1] https://github.com/guard/guard [2] - https://github.com/guard/listen

gumby2• 6 years ago

I’m sorry: why not just use make? This is a serious, not sarcastic question.

When developing I use make to invoke my program; that ensures everything is rebuilt.

anderspitman2• 6 years ago

Does make have watch functionality built in? Sometimes I use make and entr together. entr detects filesystem changes and make efficiently rebuilds. But nowadays a lot of languages have their own build tools besides make, and entr works directly with any of them.

johncoltrane• 6 years ago

Here is another way to look at it:

    $ while sleep 2; do make -s build; done

What files are interesting is already encoded in your Makefile and make is pretty good at figuring out if a given file changed or not so, instead of duplicating efforts, run make every second or two and let it figure out what needs to be done.

It won't eat your memory or pool of file pointers and your CPU will barely feel it.

gumby2• 6 years ago

Well it knows from the filesystem which files have changed.

If you mean “invoke as soon as a file has changed”: I certainly don’t want that behavior as many code changes require multiple files to be edited and there’s no automatic way to understand as the change hasn’t been written yet.

thiht• 6 years ago

Because make doesn't do this?

jchook• 6 years ago

One of the issues I've seen with entr, fswatch, etc is they don't respond to ATTR changes (e.g. via touch).

When developing on VM, sometimes you have to "forward" filesystem events to the guest. Usually that happens via TCP/UDP and touch, but wont trigger the reload.

iqandjoke• 6 years ago

any equivalent for Windows?

zaphar• 6 years ago

I created https://crates.io/crates/runwhen a while back and it's cross platform. Unfortunately you have to build it yourself :-p I never got around to creating an automated build pipeline for it.

renewiltord• 6 years ago

I used to use `inotifywait`. It met all my requirements so I just stuck to it. Allowed me to have LaTeX rendered to PDF in Evince on one side and write in Vim on the other side.

Since Evince reloads file if it changes on disk, that meant I had immediate feedback.

mixmastamyk• 6 years ago

I use a shell script named "onsave" that wraps the following incantation:

    inotifywait --event modify --recursive . --quiet

If anyone wants the full script I can post it here. One thing many of these tools miss is that I typically want the build command run once at the start.

microcolonel• 6 years ago

I will say, inotify is one of the best designed kernel facilities, it avoid so many of the pitfalls of other facilities used for the same task, and is generally easier to use. I wonder if any BSDs plan to implement it, outside of Linux ABI layers.

dang• 6 years ago

If curious see also

2017 https://news.ycombinator.com/item?id=13856623

Entr: Rerun your build when files change

Related Stories