drothlis

Claude's ability to count pixels and interact with a screen using precise coordinate

I guess you mean its "Computer use" API that can (if I understand correctly) send mouse click at specific coordinates?

I got excited thinking Claude can finally do accurate object detection, but alas no. Here's its output:

Looking at the image directly, the SPACE key appears near the bottom left of the keyboard interface, but I cannot determine its exact pixel coordinates just by looking at the image. I can see it's positioned below the letter grid and appears wider than the regular letter keys, but I apologize - I cannot reliably extract specific pixel coordinates from just viewing the screenshot.

This is 3.5 Sonnet (their most current model).

And they explicitly call out spatial reasoning as a limitation:

Claude’s spatial reasoning abilities are limited. It may struggle with tasks requiring precise localization or layouts, like reading an analog clock face or describing exact positions of chess pieces.

--https://docs.anthropic.com/en/docs/build-with-claude/vision#...

Since 2022 I occasionally dip in and test this use-case with the latest models but haven't seen much progress on the spatial reasoning. The multi-modality has been a neat addition though.

I noticed in your demo it generated the prompt "tap on the 'Log in' button located directly below the 'Facebook Password' field".

Does your model consistently get the positions right? (above, below, etc). Every time I play with ChatGPT, even GPT-4o, it can't do basic spatial reasoning. For example, here's a typical output (emphasis mine):

If YouTube is to the upper *left* of ESPN, press "Up" once, then *"Right"* to move the focus.

(I test TV apps where the input is a remote control, rather than tapping directly on the UI elements.)

Beautiful.

https://en.wikipedia.org/wiki/Characterization_test

aka snapshot tests.

According to the ViperGPT paper their "ImagePatch.find()" uses GLIP.

According to the GLIP paper,† accuracy on a test-set not seen during training is around 60% so... neat demos but whether it'll be reliable enough depends on your application.

† https://arxiv.org/abs/2206.05836

Could you implement (some of) astroid's inference using stack graphs? [1],[2]

That would allow a lot of caching optimisations, as you can "index" each file in isolation.

[1] https://github.blog/2021-12-09-introducing-stack-graphs/

[2] https://github.com/github/stack-graphs

It side-steps the problem of git conflicts, I suppose. You'd have to use their tool (`touca diff`? I don't know if that exists) instead of `git diff`.

Some ideas I got from Jeremias Rõßler's talk: https://t.co/xWtA58Q9q5

- Snapshot testing is like version-control but for the outputs rather than the inputs (source code).

- Asserts in traditional unit tests are like "block lists" specifying which changes aren't allowed. Instead, snapshot testing allows you to specify an "allow list" of acceptable differences (e.g. timestamps).

Obviously a sensationalised title, but it's a neat illustration of how you'd apply the language models of the future to real tasks.

Think systems integrators and compliance tests. I would imagine that each of the individual systems being "integrated" do have their own unit tests, upstream, in their own repos.

Some good ideas here for when your tests are in a separate repo than the system under test (GPUs/drivers/compilers in the case of the author, but it's applicable to a variety of industries).

Related: I think it was Kernighan & Pike's "The Practice Of Programming" where I read the idea of testing a complex implementation by comparing its output against a simpler but less performant implementation.

Interesting thought, somewhat related to the articles on "snapshot testing" that have been trending on HN lately.

"Regression testing" can also refer to a process: When the QA team says they're doing regression testing, it means they're testing that existing functionality hasn't regressed (as opposed to testing a new feature).

I'm not particularly wedded to any of these terms, I'm just pointing out that "regression testing" has an established meaning, and it isn't snapshot testing (outside of certain industries, at least). I do find it amusing that one implementation of snapshot testing (https://pypi.org/project/pytest-regtest/) links to https://en.wikipedia.org/wiki/Regression_testing but that article doesn't describe snapshot testing at all! Maybe the article changed? Oh well, language changes too. ¯\_(ツ)_/¯

In the article they don't change /etc/machine-id, but APT::Machine-ID in apt.conf.

https://approvaltests.com/

...and my favourite term, "characterization test": https://en.wikipedia.org/wiki/Characterization_test

"Regression test" means something else, at least at the companies I've worked at: It means a test that was written after a defect was found in production, to ensure that the same defect doesn't happen again (that the fix doesn't "regress"). It can be a manual test or an automated test. https://en.wikipedia.org/wiki/Regression_testing

I realised this is another example of "trees" as first-class citizens in a build system. In my comment above the tree we're passing around is a docker layer; in my LWN article it's an OSTree ref. We use the former for our cloud containers, the latter for our embedded device rootfs and systemd-nspawn containers.

I suppose we could use systemd-nspawn on our cloud servers too, instead of docker, but when we wrote the build system we were already using docker so it was the expedient thing to do at the time.

I suppose they're ideal for cases where there's a simple, obviously correct but slow implementation that you can use to test a more complex but faster implementation.

Fixed your link to the full article: https://lwn.net/Articles/821367/

OSTree is still working very well for us. At the time I wrote the article we had been using OSTree (and the build system I described in the article) for 4 years; 3 years later not much has changed. In those 3 years we have started building images for different architectures and our build system / OSTree handled it just fine, as you'd expect (it already handled cross-compilation, but for a single architecture).

Our build system also builds Docker containers (for our cloud services) but instead of a Dockerfile we use a custom yaml format that lists the base image (like "FROM" in a Dockerfile), the apt packages to install, and the commands to run (like "RUN"). Then we create a lockfile of all the apt packages, download them (into a local OSTree repository, but that's an implementation detail), and install them with a custom "docker run" command + "docker commit". We end up with the base layer + the apt layer + a single layer produced by concatenating all the RUN commands + a layer with our compiled Rust binaries and Python files. We use apt2ostree to generate the lockfile (really it's our patched version of aptly doing the work) but we use docker (not OSTree) to build the Docker layers. We use Docker's standard push/pull mechanisms to deploy these containers.

To hook this up to Ninja we use "marker files" (a file in the build directory) to track whether this work has been done (e.g. you need to regenerate the apt layer if that layer's "marker file" is older than the lockfile).

Our Python "configure" script (i.e. our build system) looks something like this:

    def build_docker_container(name):
        yamlfile = f"{name}/docker-base-image.yaml"
        with copen(yamlfile) as f:
            data = yaml.safe_load(f)
        ubuntu_version = data["ubuntu_version"]
        image = docker_pull("ubuntu:{ubuntu_version}")
        image = docker_apt_install(image, data.get("apt_dependencies"),
                                   lockfile="%s/Packages-%s-amd64.lock" % (
                                        name, ubuntu_version),
                                   ubuntu_version=ubuntu_version)
        cmd = "..."  # RUN commands from yaml
        deps = [...]  # dependencies from files listed in yaml
        return docker_mod(image, cmd, deps)

What I really like about the Python+Ninja combo is that you can pass these targets around as Python variables — you don't have to come up with an explicit filename for each one (the target name is generated by each helper function, e.g. `docker_pull` returns "_build/docker/${name}"). This makes it so convenient to compose these build rules. And if you ever need to debug your build system, everything is very explicit in the generated ninja file.

We don't have a ton of these containers so it has worked well enough for us because the apt layer changes rarely, and the layer above it is small/fast. The main thing we get (that you don't get from vanilla docker/apt) is lockfiles and reproducibility.

P.S. They're also great for writing tests for new code, for many of the reasons described in the OP.

I'm not the OP but I think Fibonacci is a contrived example. In practice these "characterisation tests" are great for adding tests to an existing codebase. The "snapshot" just records the current behaviour; it doesn't judge whether it's correct or not, but if you refactor it will catch changes to the behaviour, and then it's up to a human to judge if the changes were desirable or not.

I have been using this type of "expect" tests (I use the term "characterisation tests") more and more. Some examples:

- Server-side rendering and testability: https://david.rothlis.net/server-side-rendering-good/

- Testing "Page Objects" (classes written for UI tests): https://david.rothlis.net/pageobjects/characterisation-tests

I love Python + Ninja! I was going to link to my article on Ninja + OSTree, but I found in your blog you've already seen it. :-)

They probably mean incremental builds, where the actual compilation doesn't overshadow the "coordination" work.

In my (artificial) benchmark, make scaled poorly, taking 70 seconds to process 100k C files worth of dependencies, vs. ninja's 1.5 seconds: https://david.rothlis.net/ninja-benchmark/

Most of make's time was spent processing the ".d" files containing header dependencies (Ninja has a special optimisation for these files, where it reads them the first time they're created, inserts the dependency information into a binary database, then deletes them so it doesn't have to parse them in future invocations).

In real world projects, you often end up "abusing" make to add behaviour such as detecting if the compilation flags have changed, and this can make your makefiles slower; whereas ninja has those features built in. Apparently this made a big difference in build times for Chromium (where ninja was born). See this comment by the ninja's author: https://news.ycombinator.com/item?id=23182469

I found that Ninja tries hard to build things in the order they’re listed in the build file (dependencies permitting). Whereas GNU Make... well, it does start off trying to build depth-first, but if it needs to build a target's prerequisites, it pushes that target to the very end of the queue. So Make ends up building the leaf targets at the very end -- it leaves all the linking steps until all the object files have been compiled, instead of linking each executable as soon as its own dependencies are available.

I reported that here: https://lists.gnu.org/archive/html/help-make/2016-11/msg0000...

...and the maintainer said that the behaviour isn't intentional, but after a short look at the source code I decided fixing it was going to be beyond my abilities/motivation.

(I suppose this is more relevant to full builds, not incremental ones.)

Fine-grained access tokens are available on org-owned repos too, but the org has to opt in (for some reason).

It can't even do arithmetic.

Reminds me of natbot, which uses GPT-3 to do all 3: https://github.com/nat/natbot

Related, "natbot" uses the stock GPT-3 model (no fine-tuning apart from the examples in the prompt) to drive a browser:

https://github.com/nat/natbot