rckrd

Interesting that none of the new features (DALLE-3, Advanced Data Analysis, Browse with Bing) are usable without enabling history (and therefore, using your data for training).

I've also compiled a list of leaked system prompts from various applications.

[0] https://matt-rickard.com/a-list-of-leaked-system-prompts

Logit-bias guidance goes a long way -- LLM structure for regex, context-free grammars, categorization, and typed construction. I'm working on a hosted and model-agnostic version of this with thiggle

[0] https://thiggle.com

We use a similar trick and expose it via an API. Much easier to parse when you can guarantee the shape of the output

[0] https://thiggle.com/

We've found the same. A lot of usage through our LLM Categorization endpoint. The toughest problem was actually constraining the model to only output valid categories and not hallucinate new ones. And to only return one for single-classification (or multiple if that's the mode).

[0] https://matt-rickard.com/categorization-and-classification-w...

In more impressive news, "38% of code generated by GPT-4 does not contain API misuses"

I also released a hosted version of my open-source libraries ReLLM and ParserLLM that already supports APIs for

* Regex completion for LLMs

* Context-free Grammar completion for LLMs

https://thiggle.com/

[0] https://github.com/r2d4/rellm

[1] https://github.com/r2d4/parserllm

[2] https://github.com/thiggle/api

There's also another API on Thiggle that I've build that supports classification via a similar logit-based strategy.

I just released a zero-shot classification API built on LLMs https://github.com/thiggle/api. It always returns structured JSON and only the relevant categories/classes out of the ones you provide.

LLMs are excellent reasoning engines. But nudging them to the desired output is challenging. They might return categories outside the ones that you determined. They might return multiple categories when you only want one (or the opposite — a single category when you want multiple). Even if you steer the AI toward the correct answer, parsing the output can be difficult. Asking the LLM to output structure data works 80% of the time. But the 20% of the time that your code parses the response fails takes up 99% of your time and is unacceptable for most real-world use cases.

[0] https://twitter.com/mattrickard/status/1678603390337822722

Yep -- matt (at) matt-rickard.com

Thank you! I'm waiting to write this post (I follow Patrick Collison's advice methodology -- wait 10 years before you can accurately reflect [0]).

But here's Marc Andreessen thoughts:

"Seek to be a double/triple/quadruple threat."

He talks about the MBA + Undergrad Engineering combo in this blog post. https://fictivekin.github.io/pmarchive-jekyll/guide_to_caree...

[0] https://patrickcollison.com/advice

It used to be self-hosted, but I recently moved the list to Substack to make it easier for readers if they have an existing Substack account.

Thank you!

https://matt-rickard.com

779 blog posts. Writing about engineering, startups, math, and AI.

Many of the posts have rich discussions on HN. You can see the top ones here: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

---

* Reflections on 10k Hours of Programming (421 points) - https://news.ycombinator.com/item?id=28086836

* Don't Use Kubernetes Yet (306 points) - https://news.ycombinator.com/item?id=31795160

* Google search's death by a thousand cuts (292 points) - https://news.ycombinator.com/item?id=36564042

* The Unreasonable Effectiveness of Makefiles (256 points) - https://news.ycombinator.com/item?id=32438616

* I Miss the Programmable Web (248 points) - https://news.ycombinator.com/item?id=32284375

* What Comes After Git? (227 points) - https://news.ycombinator.com/item?id=31984450

---

RSS Feed: https://matt-rickard.com/rss

Email list: https://matt-rickard.com/subscribe

For a less dramatic strategy with LLMs that expose the tokenizer vocabulary, you can use context-free grammars to constrain the logits according to the parser so that the LLMs only generate valid next tokens for the language.[0]

[0]https://github.com/r2d4/parserllm

Here's mine that runs entirely in the browser and doesn't send any data to a server[1]

[1] https://chat.matt-rickard.com

There's probably a better API that wraps generate, but there's a bit more work than the logit mask.

You have to go one token at a time, otherwise the masking becomes combinatoric rather than linear (two tokens at a time -- need to generate all two token pairs, etc.).

But otherwise, that's what the code does! https://github.com/r2d4/rellm/blob/main/rellm/rellm.py#L21

(author here) That's interesting! Maybe there's a way to quantify the cumulative probability of the squashed tokens (i.e., if you constrain to 'true' and 'false', what's the distribution of the other tokens).

For now, this is a good way to make sure that I can parse the output reliably in the minimal amount of completions (instead of looping until conformant).

Similar strategies with the logitsprocesor. It is a more generalized version that's not just constrained to JSON parsing, but any regex. JSONformer/clownfish try to parse the types syntactically.

A regex is a better fit for a different class of problems. You might implement a JSONformer/clownfish with this instead.

Maybe images are the universal interface. With some of the advancements in ML, we have different decoders: image-to-text (OCR), layout information (object recognition), and other metadata (formatting, fonts, etc.).

Now, with diffusion-based models like Stable Diffusion and DALL-E, we have an encoder – text-to-image.

Natural analogy to how humans perceive the world and how we've designed our own human-computer interfaces.

[0] https://matt-rickard.com/screenshots-as-the-universal-api [1] https://twitter.com/mattrickard/status/1577321709350268928

Large fixed costs, economies of scale, whole product (need compute AND storage). It's possible that a company like Cloudflare can disrupt certain verticals in storage that are mispriced by AWS yet have a larger than expected TAM.

A bit tangential, but in some cases you can use S3 Gateway endpoints, which can dramatically reduce your egress costs if you're just accessing it within a VPC.

[0] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpo...

Those blog posts are a good start. I was referencing the full pricing page for s3 that breaks down by region and class. AFAIK there are some third parties that track the granular data, but it's not preserved anywhere on the AWS page. A lot of price decreases also might have happened silently (without a blog post) or as de facto decreases (widely but privately negotiated, e.g. "sticker price").

I still think that CUE/other configuration languages are too complex for most developers. Once you're adding language-like features (inheritance, schema validation, or control flow), you're better off using a general purpose language like Typescript.

Pulumi might be moving away from pure Typescript constructs because it's difficult to compete against something like AWS CDK, which will always be better because (1) it's specific to one provider and (2) is a first-class citizen.

I can't find the code anymore but I originally saw this in docker/compose.

I forget what they were using it for, but it boiled down to packing up a json file or two in a tarball and pushing it to the local daemon with a special suffix. Then it could pull that and unpack it on the next run.

It actually does nowadays. The --platform flag integrates with QEMU. It's emulation so it's slower, but better than doing it manually.

[0] https://docs.docker.com/build/buildx/#build-multi-platform-i...

Essentially the same thing in one step. Can avoid some of the permission issue gotchas with `docker cp`.

Docker just announced Docker Extensions for Docker Desktop -- already configured long-running agents that are "managed" with a nice GUI.

I use the Tailscale extension to let my development pods talk to my tailnet. It's really handy (would be even better if it worked with the local Kubernetes cluster in Desktop). Of course you can do this yourself fairly easily, but it's nice to have it "managed" for a non-essential part of the workflow.

I've been playing around with changing the default shell for some Makefiles to a shell inside a docker container (with dependencies baked in). It's a bit hacky, but kind of interesting!

[0] https://www.gnu.org/software/make/manual/html_node/Choosing-...

Sorry, I should I have linked some in the article.

- Alternative docker frontends - https://matt-rickard.com/building-a-new-dockerfile-frontend/ (https://github.com/r2d4/mockerfile)

- Docker merge - https://matt-rickard.com/docker-merge/ (https://github.com/r2d4/docker-merge)

This is a classic merger arbitrage spread.

[0] https://en.wikipedia.org/wiki/Risk_arbitrage