jasim

We might be moving towards putting more intent - in text - into the code itself, for agents. We should've been doing that for other humans in the past, but it wasn't always encouraged - "code should be self-explanatory, and comments are a sign of poor naming and abstraction", and all that.

These days I run a prompt of this shape to ensure that my code contains as much tacit knowledge as I have -- "I want to make sure all the implicit invariants and structures and concepts described in these plans, are present in the code comments, namings, comments around major types and modules. So even if the original plan file is not present, we should still be able to understand it (Peter Naur - Programming is Theory Building)"

However - I still need the type system as another preserver of invariants and intents. Coding agents still make subtle mistakes and miss very obvious things as the code gets more intricate and conceptually large.

A type error where an expected case is not handled - that is a feedback that can ground it back and force it to think through whether windspeed matters to is_dangerous or not. And the choice of is_dangerous in the article is a bit misleading, because anyone can at first read make the connection from windspeed to is_dangerous. It is a pretty bold variable name and attention grabbing situation.

However codebases are made of far more mundane but widespread and braided sets of tiny invariants that wouldn't catch either a human's or a coding agent's attention without needing a more firmer hand, and so I still do love my type systems despite my best laid comments.

But couple that with a deterministic system - the type checker with exhaustive destructing (and pattern matching, while we are it it) - that would become a fool-proof combination.

I hadn't considered that, you are right. Though I would still prefer TS because the language is all about types; the entire ecosystem is well-typed and the type system is quite powerful (I do enjoy an occasional Omit and Pick). But it is a personal choice; as long as LLMs can generate well-typed Python, then for people who like the language it makes sense. However from the article, I got the sense that they went completely dynamic.

I'm curious about the choice of Python, rather than TypeScript.

I find Ruby a very beautiful language, and Rails is an excellent web framework, but I need typed functions, record types and sum types.

They help not just with correctness, but also as living documentation that lets me understand AI generated code. TypeScript provides discriminated union, but not exhaustive pattern-matching, and its syntax is a bit verbose, but since I'm no longer writing most of the code myself, I can live with it.

However I can't imagine using Python or any other dynamic language going forward. There is likely good reason for you to choose it, and I'm curious to know what that is.

There is no other stack in the last 15 years or so that I look back with the same fond affection as building business applications in Clipper. One important aspect was that software was delivered physically, and I could watch my users using them. If I made a report that helped the operator, or made a workflow easier, their happiness was immediate and nurturing. And there was an immediacy to programming in xBase - if we're building a database backed business application, which was their strong suit, there was very little standing in between us and the problem space.

"There was no exploit. No vulnerability disclosure. No CVE for me to write. The attacker filled out my signup form 942 times, made 942 workspaces, sent 942 batches of about a hundred invitations each, and stopped. They used my tool exactly as designed. The design was just bad enough that the tool was good for phishing."

In my early programming days, working with Clipper, I used to look at Delphi from a distance with awe and a bit of jealousy. There also used to be PowerBuilder and Paradox, as competition to the xBase platforms.

I'd love to hear more about how you're using Delphi and what it excels at, compared to current web and native software stacks.

Everytime someone mentions Clipper (or dBase or FoxPro, or even FoxBase, but Clipper the most), I feel a sene of productive nostalgia, and a constructive anger at the state of technology today. xBase was a beautiful thing - I haven't had as much fun at building software that I've had from the first plink86 till CA-Clipper 5.3's blinker and exospace. Even prolific use of Opus 4.6 doesn't bring the sense of quality and satisfaction that those systems produced.

I'm building a new database tool for the web, a frankenstein of Lotus 1-2-3, dBase, MS-Access, and Claude Code. It is where that anger goes these days.

I'm not sure if this is what you're looking for, but there is a proposal in DataFusion to allow user defined types. https://github.com/apache/datafusion/issues/12644

Parquet files include a field called key_value_metadata in the FileMetadata structure; it sits in the footer of the file. See: https://github.com/apache/parquet-format/blob/master/src/mai...

The technique described in the article, seems to use this key-value pair to store pointers to the additional metadata (in this case a distinct index) embedded in the file. Note that we can embed arbitrary binary data in the Parquet file between each data page. This is perfectly valid since all Parquet readers rely on the exact offsets to the data pages specified in the footer.

This means that DataFusion does not need to specify how the metadata is interpreted. It is already well specified as part of the Parquet file format itself. DataFusion is an independent project -- it is a query execution engine for OLAP / columnar data, which can take in SQL statements, build query plan, optimize them, and execute. It is an embeddable runtime with numerous ways to extend it by the host program. Parquet is a file format supported by DataFusion because it is one of the most popular ways of storing data in a columnar way in object storages like S3.

Note that the readers of Parquet need to be aware of any metadata to exploit it. But if not, nothing changes - as long as we're embedding only supplementary information like indices or bloom filters, a reader can still continue working with the columnar data in Parquet as it used to; it is just that it won't be able to take advantage of the additional metadata.

I think this post is a response to some new file format initiatives, based on the criticism that the Parquet file format is showing its age.

One of the arguments is that there is no standardized way to extend Parquet with new kinds of metadata (like statistical summaries, HyperLogLog etc.)

This post was written by the DataFusion folks, who have shown a clever way to do this without breaking backward compatibility with existing readers.

They have inserted arbitrary data between footer and data pages, which other readers will ignore. But query engines like DataFusion can exploit it. They embed a new index to the .parquet file, and use that to improve query performance.

In this specific instance, they add an index with all the distinct values of a column. Then they extend the DataFusion query engine to exploit that so that queries like `WHERE nation = 'Singapore'` can use that index to figure out whether the value exists in that .parquet file without having to scan the data pages (which is already optimized because there is a min-max filter to avoid scanning the entire dataset).

Also in general this is a really good deep dive into columnar data storage.

Well put. It should always be "Created by <person>" rather than "Processed by LLM". We can already see it with Claude Code - its commit messages contain a "Generated by Claude Code" line, and it guarantees a pandemic of diffused responsibility in software engineering. But I think there is no point in railing against it - market forces, corporate incentives, and tragedy of the commons all together make it an inevitability.

I'm sorry I don't follow. The fact that you use an LLM to classify a transaction does not mean there is no audit trail for the fact. There should also be a manual verifier who's ultimately responsible for the entries, so that we do not abdicate responsibility to black boxes.

Accounting, specifically book-keeping, really plays to the strengths of LLMs - pattern matching within a bounded context.

The primary task in book-keeping is to classify transactions (from expense vouchers, bank transactions, sales and purchase invoices and so on) and slot them into the Chart of Accounts of the business.

LLMs can already do this well without any domain/business specific context. For example - a fuel entry is so obvious that they can match it into a similar sounding account in the CoA.

And for others where human discretion is required, we can add a line of instruction in the prompt, and that classification is permanently encoded. A large chunk of these kind of entries are repetitive in nature, and so each such custom instruction is a long-term automation.

You might have not been speaking about simple book-keeping. If so, I'm curious to learn.

I can think of two instances, where the LLM embracing best practices for human thought leads to better results.

Claude Code breaks down large implementations to simpler TODOs, and produces far better code than single-shot prompts. There is something about problem decomposition that works well no matter whether it is in mathematics, LLMs, or software engineers.

The decomposition also shows a split between planning and execution. Doing them separately somehow provides the LLM more cognitive space to think.

Another example is CHASE-SQL. This is one of the top approaches in Text-to-SQL benchmark in bird-bench. They take a human textual data requirement, and instead of directly asking the LLM to generate a SQL query, they run it through multiple passes: generating portions of the requirement as pseudo-SQL fragments using independent LLM calls, combining them, then using a separate ranking agent to find the best one. Additional agents like a fixer to fix invalid SQL are also used.

What could've been done with a single direct LLM query is instead broken down into multiple stages. What was implicit (find the best query) is made explicit. And from how well it performs, it is clear that articulating fuzzy thoughts and requirements into explicit smaller clearer steps works as well for LLMs as it does for humans.

With negative/positive, the invariant would be sum(amount) = 0; with your approach, it would be sum(debit-credit)=0. Both are valid, it is just two ways of expressing the same thing.

I think it is useful to think about double-entry book-keeping in two layers. One is the base primitive of the journal - where each transaction has a set of debits and credits to different accounts, which all total to 0.

Then above that there is the chart of accounts, and how real-world transactions are modelled. For an engineer, to build the base primitive, we only need a simple schema for accounts and transactions. You can use either amount (+/-ve), or debit/credit for each line item.

Then if you're building the application layer which creates entries, like your top-up example, then you also need to know how to _structure_ those entries. If you have a transfer between two customer accounts, then you debit the one who's receiving the money (because assets are marked on the debit side) and credit the other (because liabilities are on the credit side). If you receive payment, then cash is debited (due to assets), and the income account is credited (because income balances are on the credit side).

However, all of this has nothing to do with how we structure the fundamental primitive of the journalling system. It is just a list of accounts, and then a list of transactions, where each transaction has a set of accounts that get either debited/credit, with the sum of the entire transaction coming to 0. That's it -- that constraint is all there is to double-entry book-keeping from a schema point.

Instead of using negative numbers, Accounts have normal balance: normal credit balance literally means that they are normal when its associated entries with type credit have a total amount that outweighs its associated entries with type debit. The reverse is true for normal debit balance.

But that is an interpretation made by the viewer. A customer typically is an asset account, whose balances are in the debit column. But if we somehow owe them money because let's say they paid us an advance, then their balance should be in the credit column. The accounting system need not bother with what the "right" place for each account is.

It is quite practical to have only a simple amount column rather than separate debit/credit columns in a database for journal entries. As long as we follow a consistent pattern in mapping user input (debit = positive, credit = negative) into the underlying tables, and the same when rendering accounting statements back, it would remain consistent and correct.

That part of the article felt quite wrong to me as well. I've built accounting systems that worked well for a decade, where internally the values were a single amount column in the journal table. If it was a debit, it'd be positive, if a credit, it'd be negative.

In fact, we could call these values yin and yang, for all it mattered.

Also, I'm not able to really follow what he means by "money = assets in the future".

Money is money, but if you wanted to track the intermediate state until the customer gets receipt, you would use an In Transit account (Good In Transit / Service In Transit etc.)

Yet, it doesn't change the fundamental definition of the value in the accounting system. I think the author confuses an engineering concept (sagas, or thunks, or delayed but introspectable/cancellable actions in general) with accounting.

Thanks, that makes sense. I avoid using LLMs for math because it is only a text token prediction system (but a magical one at that), and can't do true numeric computation. But making it write code to compute works well.

I'm curious to hear more about this. I've seen very little hallucination with mainstream LLMs where the conversation revolves around concepts that were well-represented in the training data. Most educational topics thus have been quite solid. Even asking for novel analogies between distant and unrelated topics seem to work well.

Good question. LLMs are surprisingly less fragile than hand-coded parsers for unstructured data like the ones in a bank statement.

And to be clear - I'm not sending the entire statement to Claude; instead, only the account name/narration of those transactions for which I already don't have a mapping. Claude then returns a well-formatted JSON that maps "amzn0026765260@apl" to "expenses:amazon", and "Veena Fuels" to "expenses:vehicle" and so on.

I can also pass in general instructions saying that "Restaurants and food-related accounts are categorized under 'expenses:food'", and it does a good job of mapping most of my dining out expenses to the correct account head.

The actual generation of journal entries are done by a simple Python script. The mapping used to be the hardest part, and what used to need custom classification models is just a simple prompt with LLM.

Plain Text Accounting has become significantly easier to do for me on a regular basis, thanks to LLMs. Specifically: importing bank statements into hledger and avoiding manual entry.

I use a JSON file to map bank entries to my hledger accounts. For new transactions without mappings, I run a Python script that generates a prompt for Claude. It lists my hledger accounts and asks for mappings for the new entries.

Claude returns hledger journal entries based on these mappings, which I can quickly review.

Then another script prints out hledger journal entries for that month's bank transactions, all cleanly mapped. It takes me just a few minutes to tweak and finalize.

I can also specify these mapping instructions in plain-language which would've otherwise been a fragile hodgepodge of regexps and conditionals.

It is hard to believe that a company with the expertise of Adobe in graphics programming (Photoshop, Premier, After Effects, Illustrator) cannot build a vector design program.

Adobe XD was almost there, and they made some great decisions in nooks and crannies usually forgotten - like bringing in Chrome rendering engineers to build a subset of HTML and CSS for the plugin ecosystem.

While there are hundreds of little decisions to be made - from the runtime, text rendering, gpu vs cpu rendering, to how shadows are rendered and line heights are determined - none of these are engineering problems beyond the ken of a good team of systems and graphics engineers and designers, the likes of which Adobe has in spades.

The attempt to acquire Figma for such an enormous sum itself felt like a serious decision making mistake, and the final nail in the coffin is the complete abandonment of vector design tools.

I would pay good money to read the insider account of the corporate politics inside Adobe that led to all this.

`key` informs React of the identity of an element. That helps it during the reconciliation phase -- if it knows only one `key` in a list of DOM elements has changed, then it will run the DOM updates only on that one. Similarly if the order has changed, it only needs to move its index in the parent DOM element.

But it doesn't help in the rendering phase - aka when the virtual DOM is constructed when `render` is called on the root component, and all our JSX code is executed across the component hierarchy. Virtual DOM reconciliation cost is only a part of the performance penalty, re-running the "view as a function of state" computation is another major chunk.

Thank you!

What about long arrays? Is there a mechanism where Svelte knows which element is mutated, and do fine-grained recomputation/update only the corresponding view elements?

This is the primary place where we have to go outside the framework in React. Only a large linear list has this problem -- if it was a decently balanced component tree, then we could skip updating huge swathes of it by skipping at a top level node. But it is not possible in an array. You have it iterate through each element and do a shallow comparison to see if the references have changed.

That said, it is fairly easy to escape to DOM from inside a React component with useRef and the portal pattern. Then we can write vanilla JS which makes updates to only the values it changes.

If Svelte solves this in an elegant manner, then it would be a very compelling feature over React. I'm asking here because last I checked, most of the examples in Svelte's documentation pointed to simple counters and regular objects, and I couldn't find an example of a large linear list.

I've been using Zed for a few days now, and it is really nice!

I tend to use JetBrains IDEs for development and Sublime Text for everything else, with VSCode being an "if I don't have any other choice" option.

Zed is already feeling like a great meld of IntellJ and Sublime - the features of an IDE with the lightness of Sublime, and excellent aesthetics that Atom was once known for.

Can anyone explain the deep nostalgia and longing for old DOS era software, and in particular VGA text mode interfaces?

I love staring at these screenshots and spinning up a DOSBox every now and then and going thru 1-2-3, FoxPro, WordStar and so on.

I don't know what I'm looking for from them, does anyone else have a clue?

When you boil it down like that, then all problems in the world is a people problem. They can also be a systems problem, an incentives problem, and often a training/cultural problem. These framings are all valid, and have their place. But certain framings can help solve certain situations better.

In the narrow confines of programming, a way of thinking where we structure problems as transformation of data from one shape to another and minimizing mutable state can help systems be more robust. It is not an object-oriented vs functional programming debate. It is just a good practical way of approaching problem solving.

Unfortunately that is a very high-level description bordering on woo-woo. Which is why the book discusses these ideas with concrete code examples. For a lot of us who've hit a wall with the kind of reliable systems we could build, and discovering this new approach, it is an understanding borne out of lived experience. There is a lot of tacit knowledge here that's difficult to unpack and even more difficult to transfer to others. This book, like many others, helps, but cannot be a substitute for doing the actual work of experimenting, making mistakes, and internalizing.

But all of that is moot unless folks care about it, and care about it in the right way (like not going all in on functional programming and deep copying and immutable values where tight zero-memory-copy zero-alloc zero-kernel-call code was necessary), all of which finally makes it a people problem I guess?

Functional architecture makes extensive use of advanced abstraction, to implement reusable components, and, more importantly, supple domain models that anticipate the future.

I believe Mike Sperber is talking about Haskell when he says "advanced abstraction", but most regular line-of-business software can reap significant benefits with just simple FP principles: mostly immutable data, record types, sum types, and a vocabulary of small decomposed functions that can parse, transform, and combine this data, together building to larger wholes.

You do need a language with types and first-class functions, and the most common vehicle for that currently is TypeScript. Combine that with a book like Grokking Simplicity by Eric Normand, especially the fundamental idea of separating data, computation, and action, and you have a software system that is supple, simple, and amenable to continuous growth and refactoring over a long period of time.

Grokking Simplicity, IMO, is a severely under-discussed book when it comes to FP. Its only sticking point for me is its lack of static types, but otherwise it is the first and only book that I've seen that distils the functional way of thinking for the working programmer in an accessible way, without having to resort to the all-or-nothing proposition of completely pure FP.