Large integer precision error in Bash command output rendering

https://github.com/anthropics/claude-code/issues/11506

by rrwright • 8 months ago

42 40 8 months ago

github.com

rrwright5• 8 months ago

Try it yourself: `echo '348555896224571969'`

CGamesPlay6• 8 months ago

Looks like it has to be the full tool output to be coerced:

    > Can you run this through bash: echo '348555896224571969 plus 2 is 348555896224571971'
    
     Bash(echo '348555896224571969 plus 2 is 348555896224571971')
      ⎿  348555896224571969 plus 2 is 348555896224571971

timenotwasted• 8 months ago

Claude Code v2.0.37 Haiku 4.5 · Claude Pro

run cmd echo '348555896224571969'

I'll run that echo command for you.

Bash(echo '348555896224571969') ⎿ 348555896224571970

The command output is: 348555896224571969

If I do it this way it gets the off by 1 and then fixes it when providing me the output, very interesting.

Insanity• 8 months ago

I forgot which blogpost mentioned it, but to paraphrase it states that managers won’t understand why you can’t just fix a bug like this in a few minutes like you would in traditional software.

This might be one of those cases, where the problem arises from the training set somehow.

alyxya2• 8 months ago

This seems to be a software bug and not something about model behavior, though the model is in some sense doing the wrong thing by internally evaluating what the echo command should output rather than saying what the output actually is.

Edit: Based on the above comment showing javascript numerics behavior changing, it's more like some unusual interaction with the numeric string in the bash command being interpreted as an integer and running into precision issues.

amorriscode2• 8 months ago

Merged a PR after seeing this thread so, thankfully, this was one of those bugs that you can just fix in minutes. ;)

fulafel2• 8 months ago

Smells like floating point. Python prompt:

  >>> int(float('348555896224571969'))
  348555896224571968

It just exceeds the mantissa bits of doubles:

  >>> math.log2(34855589622457196)
  54.952239550875795

JavaScript (in)famously stores all numbers as floating point resulting in silent errors also with user perceived integers, so this might be an indication that Claude Code number handling uses JS native numbers for this.

ec1096852• 8 months ago

They wrap bash with python.

fulafel2• 8 months ago

I still suspect JS. It's much harder to shoot yourself in the foot with Python. Even if you use JSON:

  >>> json.loads('{"nr": 348555896224571969}')
  {'nr': 348555896224571969}
  >>> type(_['nr'])
  <class 'int'>

mrspuratic• 8 months ago

It seems like it, but it can't be only that. A float64 representation of an int in the range 2^58 to 2^59 should be rounded to multiples of 2^6, i.e. 348555896224571968 as you found (3.48555896224571968E17) (the final digit 9 in the math.log2() expression was lost, it's 2^58 not 2^54) The unexpected output (according to the bugreport) arises from javascript, it does NOT round like everything else for reasons I don't understand. It seems to prefer rounding to arbitrary multiples of 10 in my limited testing.

CGamesPlay6• 8 months ago

Claude Code is simultaneously the most useful and lowest quality app I use. It's filled with little errors and annoyances but succeeds despite them. Not to mention the official documentation is entirely vibe-copywritten and any quality control is cursory at best.

It forcibly installs itself to ~/.local/bin. Do you already have a file at that location? Not anymore. When typing into the prompt, EACH KEYSTROKE results in the ENTIRE conversation scrollback being cleared and replayed, meaning 1 byte of new data results in kilobytes of data transferred when using Claude over SSH. The tab completion for @-mentioning is so bad it's worthless, and also async, so not even deterministic. You cannot disable their request for feedback. Apparently it lies in tool output.

It truly is a testament to the dangers of vibe coding, proudly displayed for everyone to take an example from.

ec1096852• 8 months ago

Are you sure about the one char thing? I’d expect a huge flash if that was the case.

CGamesPlay6• 8 months ago

No, I'm not sure about the precise mechanics of it, but I noticed it because of the huge flash when using it over a somewhat laggy SSH connection. It doesn't happy in all contexts. I've definitely seen it when typing into the new-ish Claude "ask questions about the plan" flow, and I've also noticed that it redraws the entire conversation history when each new line of output is presented in a long-running tool call.

resonious• 8 months ago

It happens over ssh on cellular when the history gets long. Drives me nuts as I'm a heavy claude-over-ssh-on-phone user.

1718627440• 8 months ago

Terminal emulators are fast.

DecoPerson• 8 months ago

Ha, interesting. Using Claude Code in Zed, I never encountered any of these defects.

I just open a Claude Code Thread, tell it what I want, bypass permissions (my remote is a container), and let it work. And it works wonderfully!

I guess the “integrated” part of IDE is pretty important.

CGamesPlay6• 8 months ago

Honestly, most of the problems I have with Claude Code are frontend problems, so this wouldn't surprise me. I wonder if it's possible to make an alternative CLI frontend to it.

jimbo808• 8 months ago

I use it daily for boilerplate and CRUD stuff, and have been since it came out. I honestly haven't experienced any bugs at all with it other than Anthropic server outages, etc. As far as agentic coding tools go, nothing else is close.

That being said, it's still an LLM, and LLMs are more of a liability than an asset to me. I was an early adopter and still use them heavily, but I don't attempt to use them to do important work.

o11c• 8 months ago

Does `mosh` work better than `ssh` for this?

namibj• 8 months ago

Yes.

boxed• 8 months ago

The "failed to read file"/"failed to write file" errors that are constantly being displayed is the most glaring imo. I even get it in the interactive web version of claude.

willm• 8 months ago

I'm working on a fix for the terminal UI.

https://www.youtube.com/watch?v=OGGVdPZTc8E&t=2s

CGamesPlay6• 8 months ago

Neat!

cyral• 8 months ago

Typing "348555896224571969" into your browser's dev console will also return 348555896224571970

alyxya2• 8 months ago

I don't think this is a bug specifically with Claude Code, rather it's due to Claude Code having javascript in the backend. The interesting thing to me is that the numeric string was interpreted as an integer.

rrwright5• 8 months ago

I agree with what I think you meant: it is not a bug in Claude. However "javascript on the backend" is exactly what Claude Code is! It's a node.js implementation of automation components that feed and pull from the Claude model.

I have a VERY hard time believing that they only use JSON serialization between the model and the TUI. If they're seriealizing JSON between agents, tools, or other components, then this problem is going to continue to persist for a very long time.

ethmarks• 8 months ago

Is there any significance to the number 348555896224571969? How was this bug discovered and what was the discoverer doing?

rrwright5• 8 months ago

Why did Hacker News rename the title of this post? It was originally: "Claude Code Introduces Off-by-One Errors"

Original: https://pasteboard.co/xTjaRmnkhRRo.png

Unilaterally Edited: https://pasteboard.co/rDPINchmufIF.png

sevg2• 8 months ago

Looks like mods changed the title to the title of the GitHub Issue. This from HN guidelines is probably why:

Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.

rrwright5• 8 months ago

Good catch on the guidelines. But that github issue title obviously misses the point. The whole point is that it's a silent error in Claude Code.

dragonwriter• 8 months ago

Probably for a couple reasons:

First, HN prefers the source title unless that title is misleaing clickbait.

Second, the problem is not consistently off-by-one errors, as there is a manifestation shown in the bug of an off-by-much-less-than-one error. The problem looks like a "for some reason it seems to be roundtripping numbers in text through a numeric representation which has about [perhaps exactly] the same precisions issues as float64" issue.

kazinator• 8 months ago

Note that the number is 18 digits long.

If there is a conversion to IEEE 64 bit double involved, that type is only guaranteed to record 15 decimal digits of precision, so this number cannot be represented with enough precision to recover all of its original digits.

In C implementations, this value is represented as DBL_DIG, which is typically 15 on systems with IEEE floating point.

(There is also DBL_DECIMAL_DIG which is typically 17; that's the opposite direction: how many decimal digits we need to print a double such that the exact same double can be recovered by parsing the value. DBL_DIG existed in C90, but DBL_DECIMAL_DIG didn't appear until, it looks like, C11.)

amorriscode2• 8 months ago

We merged a fix for this that'll go out in the ~next release.

Also, for clarification, this bug was only impacting the display of numbers in the TUI, not what the model sees. The model sees raw results from bash.

rrwright5• 8 months ago

This symptom was affecting display of numbers in the TUI, but the real bug/design flaw is using jq or JSON as a transport mechanism between components or between tools. JSON's number range is famously poor and it even architecture-dependent. So if you use jq or JSON to connect components or tools, you will have this same problem silently occurring elsewhere without visibility in the UI.

This is a bigger deal than it seems like! A confidence-inducing fix would include a blog post describing a top-to-bottom audit of jq/JSON used as a transport layer between tools and components. Not just a patch to the most visible problem.

dvh• 8 months ago

There are 2 hard problems in computer science. Cache invalidation, naming things, and off by 1 errors.

ahazred8ta• 8 months ago

You left out exactly-once delivery and exactly-once delivery.

Large integer precision error in Bash command output rendering

Related Stories