I’m trying to setup a nix flake to build a LaTeX paper. With some tweaks we can coerce LaTeX to generate identical output (i.e. constant date, document id, etc) on subsequent runs. So far so good.
However, I plan to profile some code in the paper, and I would like set it up such that I can import these results into the paper. Unfortunately, benchmarks are effectively impossible to reproduce bit by bit, even on consecutive runs on the same machine. This puts me in the sticky situation where a portion of my build output is not reproducible.
This leads me to my question: how do you handle files with fundamentally random contents in a nix flake? Alternatively, is trying to build the paper end-to-end the wrong approach? Should I instead aim to provide a reproducible environment which can then build the paper?
I wouldn’t try to squeeze the irreproducable bits into a framework for reproducibility.
Instead I would use “some” measure to create the measurement data, like a huge CSV with the benchmarks results in your case and use that in derivations to produce filtered and refined LaTeX tables that you can use in your actual document.
I would also create an appendix or extra document that describes the benchmark environment as best as possible, maybe even provide a source for a base image to be used during benchmarks, and other steps necessary to meassure. This way other researches might be able to at least confirm tendencies in your data.
That’s good advice. I was planning to build a docker container from the flake for the Artifact Evaluation. Maybe even a NixOS image if the journal support evaluations on bare metal.
Wouldn’t this mean I would have to commit the results in order for it to be visible to flake? My concern is that it would litter my git log with every run. Or else if I only committed the first run, or a touched version, I would always have a dirty worktree.
A secondary, minor, issue is that my workflow would feel somewhat jarring. Every time I update the latex document I would have to run nix build, but every new benchmark would have to manually be manually run with nix develop --command <command>. However with just building the “environment”, a nix run would handle both the latex document and subsequent runs (but includes LaTeX and my codebase as runtime deps). This is can easily be solved with a Makefile, so it’s not a big deal, and just an “aesthetic concern”.
Yes, the raw measurements would be in your repository. And of course you would commit it, and as you refine your environment and how you do your measurements, that data will evolve, as well as how you process this.
A physicist or astrologist doing measurements and repeating them would do the same, I hope…
I do not see a reason why the CSV representing your (last) round of measurements shouldn’t be under version control.
The way you measure and as a result your actual measuremements develop/evolve over time, similar as how a programs source code or your papers phrasing and layout will evolve/develop over time.
Maybe you even throw out a whole paragraph (or introduce one) because of how your measurements changed? Maybe you discover some perceived discrepancy and want to compare with earlier results? Luckily you have them under version control and can check back!
I initially just planned placing different versions under different modules all of which I can benchmark at once. Since all the versions were available at once, I didn’t think there was much value to vc’ing my results. On second thoughts, I realized it’s useful especially as:
It would be interesting to see how the measurement evolve because of changes to the environment (and my nix flake) as opposed to the program source code itself.
I do the exact same thing in German as well and keep confusing them…
I am talking about the person that tries to scientifically explore the world outside of earth, by measuring planet/star distances and radiation noise and stuff, not the fortunetellers…
.