Load big data by fromJSON or import?

[cathaysia]

I have a very large data set that needs to be imported(175MB). This data can be represented in nix, json or toml, so I can choose the format I like.

But considering memory and loading speed, which format is the best choice?

[getchoo]

JSON or TOML would both be preferrable to plain Nix. I don’t know where to find any numbers, but anecdotal evidence from myself and others seem to point to either (or well, specifically JSON, but I doubt there’s a difference) being faster

Parser performance JSON vs Nix

In my experience, JSON is faster than Nix. Most likely because the parser is simpler. I wrote a little benchmark, and the difference at least for simple structures is rather small, (JSON is about 1.5x faster)

[ElvishJerricco]

@cathaysia

175MB

You should very strongly try to avoid processing 175MB with Nixlang. It’s not a fast programming language. If the information doesn’t need to be available at eval-time, doing any processing at build time will be well worth it.

[waffle8946]

Nix’s JSON parser (nlohman-json) is also particularly slow, in fact almost always the slowest among all parser implementations, hence you really don’t want to fromJSON if you can help it. Not sure about the toml parser.