The Unreal Tournament 1999 package downloads the original ISO files from archive.org. The download speed is slow and it is repeated for many system upgrades. I would like to prevent the redownload of the static ISO files.
My first hunch was to split a ut1999-data package, but even this one would depend on libarchive and triggers redownloads, whenever it is changed (e.g. glibc updates).
I have read about an experimental Nix feature “Content Addressed Derivations”. The basic idea is to hash the result of the derivation build, not its inputs. I am unsure if this could prevent the redownloads. Probably not, as one has to check, that the result hash, stays the same. This can not be done by Hydra, because UT has a non-free license.
Is the redownload the price to pay for Nix’ reproducible builds or am I missing something?
For the record, this is because the application apparently doesn’t require those ISO files at runtime, after it is built.
This means that nix doesn’t consider the ISO part of the application’s runtime closure, so it’s not part of your system closure, which means that it doesn’t get a root alongside your system, and always gets deleted on garbage collect after your system build.
This is pretty common for proprietary stuff, in general. I wrote this nushell script to create a gcroot in the project directories of any projects that depend on something like this. $path is intended to be given on the CLI, it’s supposed a manually-downloaded copy of whatever I want to create a gcroot for. Should be pretty easy to reverse-engineer if you don’t want to install nushell just to run it.
Note that if you use angrr, or any other tool that cleans up your gcroots regularly, you will want to add a suitable exception for these gcroots, they usually assume that any non-well-known gcroots aren’t intentionally kept around.
Manually creating gcroots is somewhat error-prone, and won’t actually save you rebuilds if the path changes (in which case you’d have to manually create the gcroot for the new path). Rather, I would send a pull request to nixpkgs to expose unpackIso via passthru:
Then you could add ut1999.unpackIso to your config via the option system.extraDependencies. It will still get rebuilt in some cases, but probably less frequently.
@waffle8946 If we follow the Nixpkgs route, one should probably split the package into ut1999-data package, which should be a runtime dependency of the main package. The main package should symlink the data. This way the user doesn’t need to know about passthru internals. It would still redownload every time, when libarchive changes, though. So providing the ISOs as passthru might be considered.
I have added the following snippet to my configuration to prevent redownloading the ISO-files and keep them in the nix-store. The caveats mentioned by @waffle8946 apply. I am not sure if I will go down the rabbit hole of contributing to Nixpkgs.
# Keep the Unreal Tournament ISOs in the Nix-store to avoid redownloading them.
system.extraDependencies = [
(pkgs.fetchurl {
url = "https://archive.org/download/ut-goty/UT_GOTY_CD1.iso";
hash = "sha256-4YSYTKiPABxd3VIDXXbNZOJm4mx0l1Fhte1yNmx0cE8=";
})
(pkgs.fetchurl {
url = "https://archive.org/download/ut-goty/UT_GOTY_CD2.iso";
hash = "sha256-2V2O4c+VVi7gI/1UA17IgT1CdfY9GEdCMiCYbtyNANg=";
})
];
Sorry, I must have been tired the other day. Since you’re… pinning the fetchurl result, updates to libarchive should not end up re-downloading this (unless fetchurl depends on it).
I wonder if this pattern would work with requireFile. I guess there is no reason it shouldn’t. Probably a nice QOL thing to add to modules for packages that use requireFile (though, if the actual source derivation isn’t locked behind a let binding like in this case, adding the .src of a package makes more sense than duplicating the definition).