Can Chinese AI (Kimi K3) file American tax returns (TaxCalcBench)?

https://github.com/column-tax/tax-calc-bench/pull/93

by michaelrbock • 11 hours ago

11 6 11 hours ago

dghlsakjg• 11 hours ago

For those looking for the answer that don’t want to click through the entire repo: no. Kimi cannot do your American taxes, but origin doesn’t seem to matter.

It appears that no LLM can get more than about a third of returns correct. Kimi K3 is on par with Sonnet 5. They are both very bad at doing your taxes.

persedes• 10 hours ago

Glad to see a benchmark outside of programming tasks. Presumably this one was not part of the training and shows the models performance (or lack thereof) on knowledge tasks.

yieldcrv• 10 hours ago

I like these tests

In 6 months an LLM will get it to 80%

people will exclaim negatively “it was trained on the eval [to do this incredibly useful thing]”

meanwhile a problem is solved

trescenzi• 10 hours ago

Well in this case that’s a valid complaint. The tax code changes yearly. If the primary reason an llm can get through a return is because it’s been trained on some example returns then it’ll work for a year. Which I don’t disagree that’s useful if you can retrain yearly but it doesn’t mean the problem has been solved naively.

natpalmer1776• 10 hours ago

I suppose then the benchmark should be to give it numerous synthetic tax codes written with the same language and conventions as the target tax code and then have it file "tax returns" against the respective tax codes. The training to game such a benchmark then moves from "this specific tax code" to "tax codes like this"

bwfan123• 9 hours ago

Tax codes are a great example of something changing frequently, and something very nuanced. So, the training data doesnt exist for a future year's return while are rules are already changed.

So, instead of math/programming, tax codes approximate the real world better. Whereas math/programming/images/languages are frozen in time. I find this fascinating. Almost like the legal system, wherein rules are laid out in english but are almost mathematical in the sense that there is a legal/illegal judgement made for almost every action, but that judgement is not easily determined.

Can Chinese AI (Kimi K3) file American tax returns (TaxCalcBench)?

Related Stories