I sped up serde_json strings by 20%

https://purplesyringa.moe/blog/i-sped-up-serde-json-strings-by-20-percent/

by purplesyringa • 2 years ago

336 117 2 years ago

spense• 2 years ago

awesome that serde moves so quickly. i just ran across simdutf8 and realized the pr for simd-enabled uft8 parsing is coming up on 5 years:

https://github.com/rust-lang/rust/issues/68455

383• 2 years ago

https://wikipedia.org/wiki/Poe's_law

fyrn_• 2 years ago

The parent is comparing the speed of Rust std improvement to the faster pace of Serde.

zadokshi• 2 years ago

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies.

I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable.

Rust should have Jason built in.

ninkendo• 2 years ago

It has 5 dependencies, one of which is optional, and another is serde itself: https://github.com/serde-rs/json/blob/master/Cargo.toml

    indexmap = { version = "2.2.3", optional = true }
    itoa = "1.0"
    memchr = { version = "2", default-features = false }
    ryu = "1.0"
    serde = { version = "1.0.194", default-features = false }

I don’t think you’re measuring what you think you’re measuring when you say it has 3GB of dependencies. But I can’t say for sure because you don’t provide any evidence for it, you just declare it as true.

If I were to guess, I’d say you’re doing a lot of #[derive(Serialize, Deserialize)] and it’s generating tons of code (derive does code generation, after all) and you’re measuring the total size of your target directory after all of this. But this is just a guess… other commenters have shown that a simple build produces code on the order of tens of MB…

hermanradtke2• 2 years ago

Please show your work. I cannot reproduce "3gb of dependencies".

Here is my test:

Cargo.toml

   [package]
   name = "serde-test"
   version = "0.1.0"
   edition = "2021"
   
   [dependencies]
   serde = { version = "1.0.208", features = ["derive"] }
   serde_json = "1.0.127"

src/main.rs

    use serde::Deserialize;
    
    #[derive(Deserialize)]
    struct Foo {
        bar: String,
    }
    
    fn main() {
        let foo: Foo = serde_json::from_str("\"bar\": \"baz\"").unwrap();
    
        println!("{}", foo.bar);
    }

$ cargo build && cargo build --release && du -sh target

    ...

    78M target

tredre3• 2 years ago

I arrive at almost the same result as you, with 76MB.

I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage.

OP is clearly mistaken.

cedws• 2 years ago

Jesus, 78MB is still a lot for such a simple program.

383• 2 years ago

this 100%. serde is a bloated monster, its sad that its the popular JSON, because all it does is make Rust look bad in my opinion. here are some smaller options:

https://lib.rs/crates/humphrey_json

https://lib.rs/crates/rust_json

https://lib.rs/crates/sj

skitter• 2 years ago

merde_json should also be relatively small.

INGSOCIALITE• 2 years ago

can rust use the json-c library?

duped2• 2 years ago

The value prop isn't serde_json, it's automatically generated serializers and deserializers for structured data without needing an extra codegen step like with protobufs/capnproto, plus all that machinery decoupled from the actual data format you're reading.

It essentially generates a massive amount of code that you need to write anyway, at the cost of code size and compile time. And a lot of people are happy to make that trade off.

I wouldn't call that a "bloated monster" because of that. Also, none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice.

sweca2• 2 years ago

I swear the target folder for literally any project of any scale is at least several GB in size.

gnuvince• 2 years ago

I get a progress bar when I run `cargo clean` because it's so large.

aldanor• 2 years ago

For a work project, my recent cargo clean removed 90 GB.

purplesyringa2• 2 years ago

Rust should have Jason built in.

I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.)

But also, serde_json isn't large by default. I'm not sure where you are getting those numbers from. serde_json isn't large, serde isn't large. They both have very low MSRVs few other crates support, so in all truth they can't even have many dependencies.

wiseowise8• 2 years ago

I don't think this is a reasonable approach. That's just a way to introduce bloat.

Can this meme die already? The fact that out of the box install of Rust can’t parse JSON is a joke, and you know it.

IshKebab2• 2 years ago

Std definitely differs from other crates:

1. There's only one version so you can't end up with multiple copies of the crate.

2. It is precompiled, so it doesn't bloat your target directory or compile time.

3. It is able to use unstable features without using the nightly compiler.

It's a totally reasonable approach. Many other languages have JSON support in their standard libraries and it works fine. I'm not sure I'd want it, but I wouldn't say it's an obviously bad idea.

kelnos4• 2 years ago

That's just a way to introduce bloat.

I don't think "bloat" is the issue; as I'm sure you know, Rust programs only contain code for the features of the stdlib they use; if it had JSON support and it wasn't used, the linker would omit that code from the final binary. (Granted, that would make the linker work a little harder.)

More at issue is maintenance burden. Data formats come and go over time. In 10 or 20 years when JSON has faded to XML's level of relevance, the Rust stdlib team shouldn't have to continue maintaining a JSON parser.

Klonoar• 2 years ago

What kind of machine are you developing on that runs out of space that quickly…?

pornel• 2 years ago

Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug.

Anything you compile will dump gigabytes into the target folder, but that's not representative of the final product (after stripping the debug info, or at least using a toned-down verbosity setting).

hinkley2• 2 years ago

Does it need a more compact representation of its debug info?

duped2• 2 years ago

Most of your target folder isn't debug info, but stale build artifacts because Cargo doesn't do any garbage collection.

khuey• 2 years ago

Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug.

Rust relies on the linker (via -ffunction-sections and -gc-sections) to delete functions that aren't ever used but the linker isn't capable of removing the corresponding debug info.

https://github.com/rust-lang/rust/issues/56068

jwells89• 2 years ago

Built in JSON encoding/decoding is one of the things I’ve enjoyed about Swift. It’s nice when it’s not necessary to shop around for libraries for common needs like that.

throwup2383• 2 years ago

Almost nobody is shopping around Rust JSON libraries unless they need some specific feature not provided by serde and serde_json. They are the default everyone reaches for.

troad4• 2 years ago

Dependency bloat is an issue with Rust in general. The dependency trees for any meaty Rust project quickly become pretty horrifying. Auditing all these dependencies is infeasible, and my level of confidence in a lot of them is fairly low.

I worked with Rust for a few years, and with the benefit of a few years' experience, I don't think I'll be touching Rust again until the ecosystem matures a great deal (which will only come with significant corporate adoption), or if I need something for a no-std, no-deps, strictly-a-C-replacement kind of project. (Though Zig might edge out Rust for this use case once it stabilises.)

echelon• 2 years ago

The dependency trees for any meaty Rust project quickly become pretty horrifying.

s/Rust//

This is really no different from any other language.

At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum.

C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow. You're holding Rust up to a standard that nothing meets.

SkiFire132• 2 years ago

Let's be real about dependencies https://wiki.alopex.li/LetsBeRealAboutDependencies

samatman• 2 years ago

Though Zig might edge out Rust for this use case once it stabilises.

Zig has a meaningful advantage in the context of this discussion: lazy compilation. The compiler won't even semantically analyze a block of code unless the target requires that to happen.

Currently, dependencies are more eager than they really need to be, but making them just as lazy as compilation is on the roadmap. Lazy compilation means no tree shaking is needed, and it means that the actual build-graph of a program can be traced on a fine-grained level. We might not be able to audit a hundred dependencies, but auditing the actual used code from all those dependencies might be more practical.

This is well positioned to handle a common pattern: `small-lib` provides some useful stuff, but also has extensions for working with `huge-framework`, and it needs to have `huge-framework` as a dependency to do so. Currently this means that the build system will fetch `huge-framework`, but if we can get it lazy enough, even that won't have to happen unless the code consuming `small-lib` touches the `huge-framework` dependency, which won't happen unless the program itself needs `huge-framework`.

Existing build systems don't do such a great job with that kind of structure, and the culprit is eagerness.

tinrab• 2 years ago

From crates.io, `serde` is a 76.4 KiB dependency. And from what I've seen looking through the code, it's pretty minimal.

the__alchemist• 2 years ago

I just added `serde_json` to my small GUI program. It alone increased the binary size from 6Mb to 8Mb; unacceptable for what it does.

Sytten• 2 years ago

The utf-8 tricks make me very nervous since I have seen too many attacks with parser confusion. I for with serde for correctness not speed. I hope this was fuzzed all the way with a bunch of invalid utf-8 strings.

dwattttt• 2 years ago

Luckily utf-8 structure is _very_ trivial compared to the average parser. Not to say there can't be bugs, but that the internal states of a parser shouldn't be large, and can be exhaustively tested.

hinkley2• 2 years ago

This is the sort of space where I’d like to see a fuzzer.

hsbauauvhabzb• 2 years ago

Any bugs you can point to that come to mind of this class?

eesmith• 2 years ago

https://en.wikipedia.org/wiki/UTF-8#Invalid_sequences_and_er...

Many of the first UTF-8 decoders would decode these, ignoring incorrect bits and accepting overlong results. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such as NUL, slash, or quotes. Invalid UTF-8 has been used to bypass security validations in high-profile products including Microsoft's IIS web server[26] and Apache's Tomcat servlet container.[27] RFC 3629 states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."

zorked• 2 years ago

Teaching to _think_ is just as important as teaching to code, but this is seldom done

Oh, the arrogance of thinking that the other person doesn't think.

loa_in_2• 2 years ago

It isn't what author argues at all. It's about teaching how to think: that you have to do some research and it's not a step you can skip; having done your research doesn't free you from having to draw your own conclusions too. Skipping either of those steps is easy but wrong.

ramon156• 2 years ago

is seldom done

It's this part

wtetzner8• 2 years ago

I don't think there's any arrogance in the statement. It doesn't assume others don't think. It's simply observing that most blog posts and how-to articles show the final result, but not necessarily the steps that were needed to get there.

7bit• 2 years ago

I don't see what one has to do with the other.

s_Hogg• 2 years ago

Very strong jart feel about this person's blog, that was a nice read

We would need to reinvent the wheel, but this is quite neat if you think about it.

Is this real or ironic though? I read it and started laughing at the writer but the rest of the page seems quite heavy on self-deprecation

vecplane• 2 years ago

What does jart mean?

wffurr• 2 years ago

A commonly posted author: https://justine.lol/

IggleSniggle• 2 years ago

HN username for an extremely talented software engineer and software-engineering communicator, justine.lol. Probably most known around here for her cross-platform C code (cosmopolitan) and relatedly redbean, a zip file and tool that is also an executable file-server-file that hosts itself and can produce other such self-hosting cross platform executable zip file servers.

wtetzner8• 2 years ago

I think it means the approach is quite neat, not the fact that it requires reinventing the wheel.

I sped up serde_json strings by 20%

Related Stories