Load big data by fromJSON or import?

I have a very large data set that needs to be imported(175MB). This data can be represented in nix, json or toml, so I can choose the format I like.

But considering memory and loading speed, which format is the best choice?

JSON or TOML would both be preferrable to plain Nix. I don’t know where to find any numbers, but anecdotal evidence from myself and others seem to point to either (or well, specifically JSON, but I doubt there’s a difference) being faster

You should very strongly try to avoid processing 175MB with Nixlang. It’s not a fast programming language. If the information doesn’t need to be available at eval-time, doing any processing at build time will be well worth it.

3 Likes

Nix’s JSON parser (nlohman-json) is also particularly slow, in fact almost always the slowest among all parser implementations, hence you really don’t want to fromJSON if you can help it. Not sure about the toml parser.

1 Like