thesephist

Reminds me a lot of Infinite Craft (https://neal.fun/infinite-craft/).

The generative part of language models can make for really fun "single-player" games where you're really competing with the inventiveness of the language model, so there's some sense that you're playing a game with infinite hidden complexity.

Thanks for sharing Simon! I will note that by training an adapter layer between this autoencoder's embedding space and OpenAI's, it's possible to recover a significant amount of detail from text-embedding-ada-002's embeddings with this model too[0]. But as the paper author's reply in a different thread points out, their iterative refinement approach is able to recover much more detail in their research with a smaller model.

[0] https://twitter.com/thesephist/status/1698095739899974031

Definitely possible if supported by transformers.js. If I see enough folks wanting it I'll likely add it at some point.

This is definitely an interesting idea I've also pondered before. In my experience (just speaking from intuition) what's "easy" for LMs to predict often doesn't line up with our human expectations for what's "obvious". Often LLMs will learn seemingly "low information content" statistical correlations that just helps it lower its training loss.

Hey HN! Author here.

For the many Obsidian users here, wanted to share an Obsidian demo/plugin that I saw recently by Justin Smith[0] that I think faithfully carries over a lot of what I liked about this idea into the Obsidian land, complete with a semantic index w/ language models.

If you're an Obsidian user, do check out the demo. I can't take credit for any part of building it, but it's really cool to see the idea in action :)

[0] https://twitter.com/justindsmith/status/1679978286955532296

Hey HN! Fun to see this project from my college years on HN again. To clear my name, I have actually written other languages that are arguably more useful, like Oak [0] which I used to build a bunch of personal software infrastructure [1] including my current daily-driver apps for notes, longform writing, a Twitter client, a personal search engine, and a few other knickknacks.

I find building little toy interpreters and programming languages fantastically fulfilling, creatively energizing, and a really fun way to learn how such a common element of our work that's often a black box functions inside. I've written before about how to get started [2] with these kinds of little toy programming language projects if you're curious to dive in.

[0] https://oaklang.org/

[1] https://github.com/topics/oaklang

[2] https://thesephist.com/posts/pl/

This project is from a couple of years ago. Sibling comment is correct that I'm working now.

I wonder if a hacky solution may be to have some kind of intermediate model to serialize the text (whether from an image of it or the raw PDF data) into LaTeX? I imagine the LM has seen enough formulas in TeX to understand it, but in most PDFs formulas are just jumbles of letters.

That's interesting! I think GPS falls in the category of "works so reliably that I don't perceive it as technology" for me. But yes, GPS an underrated piece of infrastructure, even more impressive as it's globally available (edit: as in, US tech infra made available beyond US).

Thanks for the repo/issues link! Didn't know that resources existed.

Looks like the immediate issues got taken care of thanks to the HF team, but I'll probably pop over there in the future if I have ideas or notice things that can be improved.

No and no - fortunately Torvalds doesn't do much web dev and conversely I don't do much kernel dev :)

The code generated is most definitely related to the names/descriptions! To do this, I have to first generate the description then generate the code _from the descriptions_. The downside of this is that I can't parallelize text generation, but the upside is the code feels much more realistic. Here's the prompt I used (from that same file):

The idea here was to give the model a prompt that felt like a tutorial or some kind, and try to minimize non-Python non-ML-y code.

---

$MODEL_DESCRIPTION_FROM_EARLIER

Let\'s use this model. The basic use case takes only a few lines of Python to run the inference. Here are the first few lines.

```python

Sorry about that. I don't think I'm leaning on any super new browser/JS features, but if you share your FF version string (or an error in the console) I can try to troubleshoot what's missing!

Yep! No fine tuning. Here's the prompt I use for the description (from source, https://github.com/thesephist/modelexicon/blob/main/src/main... )

---

Proceedings of Deep Learning Advancements Conference, list of accepted deep learning models

1. [StyleGAN] StyleGAN is a generative adversarial network for style transfer between artworks. It uses a traditional GAN architecture and is trained on a dataset of 150,000 traditional and modern art. StyleGAN shows improved style transfer performance while reducing computational complexity.

2. [GPT-2] GPT-2 is a decoder-only transformer model trained on WebText, OpenAI\'s proprietary clean text corpus based on Wikipedia, Google News, Reddit, and others comprising a 2TB dataset for autoregressive training. GPT-2 demonstrates state-of-the-art performance on several language modeling and conversational tasks.

3. [$MODELNAME]

Yep, that's spot on. The overall performance is comparable to Curie, but depending on the particular task GPT-J performs better or worse (I believe empirically it's slightly better at chat and code, worse at some others).

Yep. I didn't want to have to host user-generated data (for all the perils that carries), so the sharable links work by embedding all the generated data in the link itself.

There's a pre-generated set to (1) spare my server some work and (2) showcase some output I liked. But as sibling comment noted, you can (or could) generate your own — I'm working on bringing that side back up...

The pre-generated set is hand-curated, but they are still 100% generated by the GPT-J model behind the scenes. More info -> https://github.com/thesephist/modelexicon

This point should get more visibility — embeddings are not made in the abstract; they reflect the lexicon of their training sets, and I strongly suspect word embeddings used here reflect the lexicon and word frequencies (and their meaning/usage context) in modern literature. Using them for text in the past (nearly a century back!) warrants some skepticism.

Interesting perspective! It had actually not occurred to me to take a more general non-anthropocentric perspective and I'm grateful to know there's non-human sides to semiotics/notation to think about. TIL :)

I mention this in passing in the post, but I think the various notations that exist to express finite state machines help us greatly to understand complex software systems.

One way to express FSMs is the general node-and-arrow diagram. That makes sense for simple systems like a traffic light ("red" can only go to "green", etc. etc.) But for more complex state machines, we might want to express it in a regular expression, which is mathematically equivalent and lends itself to a different kind of mental model that might be better for, say, writing a search engine. Another "notation" to represent FSMs is matrices that model state transitions (if you can go from state 1 to state 2, X[1][2] = 1, otherwise it's 0, etc.) This is useful for certain other kinds of questions like "how many steps to go from state A to state B?.

One practical use case of state machines is in parsing theory. If you're trying to parse JSON correctly to the spec, you might care about the formal grammar of JSON, which is often written in a specialized notation called BNF that describes what symbols can come after another. This is useful for implementing a parser, but to understand the grammar itself I personally find railroad diagrams [0] more intuitive. Different notations for the same idea let us be smarter in different use cases.

On a completely different track of ideas: you might be interested in the work from Dion Systems [1] which are doing some interesting research into more dynamic ways to work with source code that I think reflects a lot of the ideas I wrote about. It touches on your specific comment more directly.

[0] https://en.wikipedia.org/wiki/Syntax_diagram [1] https://dion.systems/gallery.html

Though I mention some of these in the post, I think it might be worth calling attention to some of my favorite works in this general space of interactive documents/dynamic notation/interfaces for thinking in the software medium.

- Ken Iverson's "Notation as a tool of thought"

- Bret Victor's works around the general topic of "Dynamic medium"

- D. Englebart's "Augmenting Human Intellect" — in particular, his thoughts on symbolic manipulation technologies and the cascading effects of improving notation (though he doesn't use the word "notation" very much, the ideas are there)

- Ted Chiang's many fiction works exploring topics around language and notation: "Truth of Fact, Truth of Feeling", "Story of Your Life"

- "Instrumental interaction" by Michael Beaudouin-Lafon, which is about software interfaces, but also provides a good conceptual framework for thinking about notation as interfaces as well.

Yep! They're completely procedurally generated to be random (within some anatomically possible range). It does get slow if you really crowd up the map (or you're on a large large screen) because I didn't particularly optimize for that case that much (and it's written in a slower compile-to-JS language I'm working on). If this was a part of something more serious I'd go back and optimize it more, but for fun, it was good enough for me.

To be more specific -- the slowness comes from the fact that the entire scene is redrawn (React-style) on every change or click, instead of incrementally updating the canvas. This makes things like resizing and removing speech bubbles easy but makes other things more expensive.

Hey HN! Quick weekend hack (I did a Twitter thread of me building this [0]) that seemed fun and interesting. Apparently a lot of people were having fun seeing some of the images I could create with it [1], so I thought I'd drop it here to share some fun!

Quick tutorial:

- Click anywhere to drop new humans

- Shift-click to delete the closest human

- The other buttons do what they say

In my experience, it works best on large screens when you can really put together a crowd. Check out some of the screenshots on [1] if you're stuck on mobile.

[0] https://twitter.com/thesephist/status/1449609097632964608

[1] https://twitter.com/thesephist/status/1449948181559713798

OP here. Indeed, Iverson's seminal lecture is what the title alludes to :)

It's funny that lots of people comment on the snappy UI, because there isn't actually a lot of performance magic going on here -- the frontend just loads a few-MB large gzipped JSON inverted index and uses it to do all search and display locally in the browser using pretty normal unoptimized JavaScript. No WebAssembly or anything else fancy -- just JavaScript working through a large object graph.

I think in general simple JavaScript applications can be remarkably fast these days if you're careful not to introduce unnecessary work and aware of the data structures you're using and profile + optimize judiciously :)

Ink in its current form/design has a few fatal flaws, mostly around poorly conceived syntax and semantics. I didn't notice it in design because I was new to designing a PL, but noticed them as I wrote a lot of software in Ink. These are pretty fundamental to the language so re-doing them would constitute a hugely breaking version change, but those changes are necessary I think for me to be confident telling other people to try it and use it. Some of these "flaws" I found are:

- I want real lists/arrays rather than Lua table-style data structures

- I need either pipelines (->>, |>) or class-style methods for ergonomics

- Testing and module system stories need a lot of work.

- Syntax quirks makes some kinds of nice constructs syntactically impossible or verbose (e.g. calling a fn defined on an object requires parenthesis to override the default order of operations)

I recently started hacking on a sort of "Ink 2.0" that preserves a lot of what I like about Ink (minimalism, lispy-ness, value rather than reference types, match expressions) without many of the design flaws. I might share and push that more publicly once I feel it's getting ready.

Thanks! This comment made my day. Certainly knowing which directions not to go in is a big part of my efficiency, and something I'll point to whenever I get asked that question in the future.

+1 to both Crafting Interpreters, which is probably my favorite free online text about this stuff, as well as the Interpreter Book and Compiler Book, both written for Go as the implementation language. They were all a huge help for me learning the ropes initially.

I'm usually in NYC these days, but WL is my hometown. Appreciate the offer! Maybe we'll meet up in NY instead :)

Hey HN, OP here. Ink isn't really anywhere near in a shape for any kind of "public" or "production" use. It was a toy programming language I built to (1) learn how interpreters worked and (2) build some side projects of mine, back in 2019. Since then it's grown steadily to allow me to build more complex full-stack apps, like Monocle[0]. But the truth is many parts of the language are still rough and mostly meant for personal use. So I guess what I'm saying is — please be gentle, and hopefully there are some interesting ideas in there, or at the very least, it inspires someone else to go build toy languages of their own.

If you're curious about the personal history of the project, Ink was the subject of a recent talk I gave at GopherCon EU[1]

[0] https://github.com/thesephist/monocle

[1] https://youtu.be/nNQHw_RdXvg

EDIT: If any mod is reading, the language is called "Ink" not "dot ink", though the domain is confusing. If the title could be modified that would be... swell.