Impressive work with garnix, and super weird with nix magic cache.
One suggestion: Having something other than color to distinguish the data on the plot by would be nice
Thanks, and yeah, if anyone has any ideas what we might be missing or doing wrong, that’d be very appreciated. An example magic-nix-cache workflow is here
How do you mean exactly? Like having different point types for each CI?
Yeah, or having the legend text float near the relevant series of data (sorry, I know it’s a boring nit pick, but color blindness suck)
Not a nitpick at all!
I changed the color scheme to a more color-blind friendly one, and changed the hover tooltip on the graph itself so that now it only mentions the CI you actually clicked on. I wasn’t sure how to incorporate shapes since they’re already being used to indicate errors or timeouts, but if it’s still unclear, let me know and I’ll try to change that too!
That’s super considerate. Especially the tooltip change is great, but the colors also look better (to me at least).
Do you have any numbers for how much it cost to run the tests for each platform? I guess it might be difficult to figure out if pricing structures are very different.
Agreed! Might be my new default palette!
Yeah, I wanted to write a cost breakdown, but it was going to be a decent amount of work to do accurately and not misrepresent things.
In very broad terms, speaking for how much usage running these benchmarks ran up (which is already only an approximation of real costs, since sometimes for early tests I wasn’t using all platforms, and some platforms were failing somehow and therefore not running for as long, etc), garnix and nixbuild.net were in the same ballpark. Cachix was free for 14-days, and it could be that even the forever-free (5GB) plan would have worked for our purposes. GitHub Actions is much more variable though; it can be free (for open source), but if it were in a closed-source repo, the total GitHub Actions bill would have been about 40 times more than either nixbuild.net or garnix. Granted, we ran several different CIs with that, but even if we divide it by 5 we still get a big factor more.
But if the question is more “will this be too expensive for me to rerun?” - well, I think you might get away with testing it in the limits included in all the various free plans, if your repo is not too big, public, and you’re doing around 10 commits.