Hello,
I’m wanting to package some environmental data (water current predictions) for deployment through nixops. I’m already doing this for other variables and it is working quite nicely, however in this case the input files I need to extract the required data from are quite large (~30gb).
Our current practice is to extract the data we need first (approximately 250mb), then put these files into a hg repository for incorporation into our nixops deployments. However the extraction is just a fairly mechanical extract from the large file and easily codified into a derivation - except the dependency is big.
Is there a best practice on using larger files with the nix store? I know I’ve seen warnings about memory usage when my store directories get big. Should I host the large file, download it before my build (fetchurl?) then delete the resulting nix-store entry after I have generated/instantiated the reduced derivation?
Thanks in advance,
Joe