peterohler

Well, blog entry is still there at https://orbisappsec.com/blog/critical-buffer-overflow-in-ojs... but it is total nonsense and a hallucination.

I was contacted by the submitter and they apologized and removed the blog entry. It was AI generated. It was nice to see they were upstanding enough to correct it. That's a plus in my book.

Another alternative is oj, https://github.com/ohler55/ojg. I don't know how the performance compares to jq or any others but it does use JSONPath as the query language. It has a few other options for making nicely formatted JSON and colorizing JSON.

I've been writing Lisp code off and one since the 80s. The standard for Common Lisp has to be sbcl but the REPL is pretty minimal. The available packages tend to be more limited than Go which I've been using a lot lately. I did find a way to have a more functional REPL and also have access to all the Go packages by writing SLIP (https://github.com/ohler55/slip). Yes I know this is a plug for SLIP and if that offends anyone I apologize. The reasons mentioned for developing it are valid though and I've managed to use Lisp for almost all the data mining and processing tasks.

Over the last 2 or 3 years I've been building a Common LISP implementation in Go so that Go packages can be utilized by LISP code. Building a REPL with lots of interactive features was rewarding as was taking up the challenges of the object systems (CLOS and Flavors) and generics. Just open sourced it at the start of January. https://github.com/ohler55/slip

Thanks for the help.

After several years of development here is a mostly Common LISP implementation written in Go with a REPL, CLOS, generics, Flavors, and much more.

If you prefer JSONPath as a query language, oj from https://github.com/ohler55/ojg provides that functionality. It can also be installed with brew. (disclaimer, I'm the author of OjG)

I'm clearly biased but oj which uses JSONPath is my preferred JSON manipulator. It can be installed with brew. It may not be for everyone but some of you might like it.

My experience is quite a bit different. Of course the examples I would use are more like what you might expect in real code. The comparison should be against code that calls a function that either returns and error and checks that error or one that panics and recovers. The overhead of returning the extra error and then the conditional used to check that error is more than a panic on error and recovery somewhere up the stack. This was not true in the early days of go but it is true today.

It really depends on the code being written. Try one approach then the other and see if it works better in your situation. For the example in the article there is really no need for an error check in the idiomatic case so why compare that to using panic. If there was an error to check the result would be much different.

Merged. Didn't seem to make much difference though. Results for the original Oj parser are pretty close to the core json now. I'll have to update the README for Oj. It's a bit stale. The new Oj::Parser is still much faster if not restricted to the current Rails environment.

If you would like to discuss separately on a call or chats I'd be up for that. Maybe kick around a few ideas.

I missed responding to your assertion that the Oj::Parser was not thread safe. An individual Oj::Parser instance is not thread safe just like other Ruby object such as a Hash but multiple Oj::Parser instances can be created in as many threads as desired. The reason each individual Oj::Parser is not thread safe is that it stores the parser state.

Just so you know, I am impressed by the depth you've delved into with JSON parsing and dumping. It looks like a lot of time and effort went into the analysis.

The strict mode benchmarks for Oj are in the test/perf_strict.rb. Others are are in perf_*.rb.

If callback parsing is not supported that's fine. Oj does support callback parsing as it allows elements in a JSON to be ignored. That save memory, GC, and performance. Your choice of course just as including callback parsers is a choice for Oj.

Ok, so you picked options that you knew would fail. Again you choice but there are certainly others that would trade a slight improvement in performance to not have 16+ significant digits. It's a choice. You are certainly entitled to you opinion but that doesn't mean everyone will share them.

I'm not sure what platform you are testing on but i'm sure there will be variations depending on the OS and the hardware. I tested on MacOS M1.

Using the benchmarks in the Oj test directory Oj has a slight advantage over the core json for dumping but not enough to make much difference. The comparison for Oj strict parsing compared to the core json is more substantial as 1.37 times faster. The benchmarks use a hash of mixed types included some nested elements.

The callback parsers (Saj and Scp) also show a performance advantage as does the most recent Oj::Parser.

As for the dumping of floats that are at the edge of precision (16 places), Oj does round to to 15 places if the last 4 of a 16 digit float is "0001" or "9999" if the float precision is not set to zero. That is intentional. If that is not the desired behavior and the Ruby conversion is preferred then setting the float precision to zero will not round. You picked the wrong options for your example.

I would like to say that the core json has a come a very long way since Oj was created and is now outstanding. If the JSON gem had started out where it is now I doubt I would have bothered writing Oj.

Oj author here. While it's flattering to have Oj be the standard to beat I'd like to point out that most of the issues with Oj revolve around the JSON gem and Rails doing a monkey patch dance and Oj trying to keep pace with the changes. The Oj.mimic_JSON attempts to replace the JSON gem and only replaces the monkey patches made by that gem. The preferred approach for Oj outside of trying to mimic the JSON gem to to never monkey patch. That approach is used in all other modes that are not mimicking the JSON gem or Rails. I should point out that other Oj modes perform much better than the JSON gem and Rails modes.

There are a few more tolerant versions of JSON. In OjG I called the format SEN https://github.com/ohler55/ojg/blob/develop/sen.md

Very cool. I don't know if it's too much of an ask but could you adopt that to also work with OjG which uses JSONPath for instead of the jq syntax. I'd be glad to help if you are up for it. My apologies if I am out of line.

I found Jq to be difficult to use which is why Oj, https://github.com/ohler55/ojg is based on JSONPath. There still are a lot of options but it only takes a couple of help screens to figure out what the options are.

You might want to take a look at https://github.com/ohler55/ojg. It takes a different approach with a single pass parser. There are some performance benchmarks included on the README.md landing page.

The Oj JSON parser and data extractor is now available on brew. The oj application is similar to jq but relies on JSONPath instead of the jq language.

brew tap ohler55/tap brew install oj

For some specific cases simdjson might be the fastest but it does fare as well in other cases as seen here: https://github.com/ohler55/compare-go-json or here: https://github.com/ohler55/ojc/blob/master/misc/compete.md. Parsing the JSON file is only part of the solution though. There needs to be an efficient and easy way to extract the data of interest. OjG (https://github.com/ohler55/ojg) has a JSONPath implementation for that. Written in go is a high performer.

For me OjG (https://github.com/ohler55/ojg) has been great. I regularly use it on files that can not be loaded into memory. The best JSON file format for multiple record is one JSON document per record all in the same file. OjG doesn't care if they are on different lines. It is fast (https://github.com/ohler55/compare-go-json) and uses a fairly complete JSONPath implementation for searches. Similar to jq but using JSONPath instead of a proprietary query language.

I am biased though as I wrote OjG to handle what other tools were not able to do.

Electric stoves are fine until the power goes out. With power issue in California it's nice to be able to cook a meal on the stove instead of resorting to a tiny camp stove.

Another alternative is the oj app (ojg/cmd/oj) which is part of https://github.com/ohler55/ojg. It relies on JSONPath for extraction and manipulation of JSON.

A mongo database can have multiple collection or tables. Data from multiple tables can be joined together and then just what the user asks for is returned. We also include fields that calculate values based on data from the database. The caller need not be aware of the layout or structure of the data in the database. GraphQL queries allow for just the requested data to be returned and not a complete record from a database.

It's worked out great for us. We use it as a flexible way to get exactly what is needed in a single request that ends up pulling data from a mongo database. It has not added any extra complexity. On the contrary it has simplified out backend and front end apps. We use go and https://github.com/UHN/ggql.

Just tried [OjG](https://github.com/ohler55/ojg) and saw anywhere from zero to 20% faster on the various benchmarks. Nice improvement overall. Hats off to the golang maintainers.

It actually started with XML. At the time there were not any decent options so I wrote my own and then made it a C extension. JSON can next since XML worked out so well. Basically JSON parsers were slow and I thought I could do better.

It started out as just a need for better performance than the standards but with golang I felt a full implementation of JSONPath would be of benefit to myself and others.

The real motivation though is the challenge and knowing people are using the code I write.