Creating vendor directories directly in srcs of go and rust packages so fixed output derivations won't be needed

So there’s https://github.com/NixOS/nix/issues/2270, and https://github.com/NixOS/nixpkgs/issues/84826 which suggest that it was a bad decision to design buildGoModule and rustPlatform.buildRustPackages as they are today. I’m not sure I deeply understand the core issue of https://github.com/NixOS/nix/issues/2270, but I understand that if we could only go back in time and avoid fixed output derivations for sources, we’d satisfy edolstra.

It’s commonly known, that the alternative to buildGoModule is buildGoPackage which doesn’t produce fixed output derivations but people don’t like to use it because it requires adding a deps.nix that is large in size and it’s not comfortable to update it.

I have an idea but I wonder if it’s too wild: Rewrite buildGoModule so it will parse (in Nix) go.sum from ${src} and make it construct the fetchurl or fetchgit calls that will satisfy the check sums found there. Emulating Go’s hash calculation in Nix wouldn’t be easy (these are not exactly plain sha256), but I’ve found this project https://github.com/vikyd/go-checksum which claims it has implemented this, in Go.

As for rewriting buildRustPackage in a similar manner, it seems there’s a somewhat complete Nix “Parser” and hashes calculator - https://github.com/nmattia/naersk/blob/a82fd7dc31a58c462b6dfa9d9d886fa2cc75dfd4/build.nix .

Is it possible to read a file found in ${src} and do pure Nix stuff based on it’s contents?

I think edolstra added the fromTOML builtin specifically for this usecase; as nix has a Rust dependency now.

See https://github.com/NixOS/nix/blob/8351d36b214fcd00f4a769e96ebaeb22af41164c/release.nix#L17-L24

For that to work we need to include the go.sum of every package into nixpkgs.
Also we need to modify buildGoModule to reconstruct the modules.txt inside the vendor directory.
Importing $src does not scale because that would means that we need to import each time we evaluate nixpkgs every package we have. We do not allow this in nixpkgs for the same reason.
Since we need to add extra metadata anyway I would actually generate nix source code directly since parsing other file formats with nix without builtins is potentially slow. Maybe we can make the tooling for this better than go2nix so that updating it, does not suck.

I see. There’s another topic which seems related now, which is “importing from derivation” which hydra doesn’t yet support: https://nixos.wiki/wiki/Import_From_Derivation & Is importing a derivation from another repository bad practice?

If hydra was able to import from a derivation, it would have been possible to create a deps.nix inside the $src of go modules projects, and then import it?

Yes. But also this would be expensive. It would means that if you for example want to install docker from the binary cache, you would first need to download the docker source code to do the import from derivation.

It would mean that if you for example want to install docker from the binary cache, you would first need to download the docker source code to do the import from derivation.

WOW … :sweat_smile:

It feels pretty insane that these ecosystems have worked so hard to make sources reproducible but we can’t use their reproducibility methods directly in Nix? Maybe this observation is incorrect, because with either of the methods I mentioned above, it will be reproducible but it will require either downloading the source to evaluate (with Import from derivation) or downloading go.sum files to Nixpkgs…

You’ve mentioned:

Also we need to modify buildGoModule to reconstruct the modules.txt inside the vendor directory.

I noticed that in your recent improvements to buildGoModule, you linked the go-modules derivation to ./vendor. I wonder: Is it possible to use an extraPostFetch instead? This will spare one hash from the buildGoModule derivations.

Maybe Internet is not available for commands in this hook but it’s not documented so IDK. If so, then perhaps we need to make fetchurl and friends be able to perform arbitrary actions with internet access.

This sounds feasible.

1 Like

:upside_down_face: The question that remains is: Such srcs wouldn’t be considered by edolstra as fixed output derivations in disguise right?

No. I think his plan is to move all fetchers over to builtins. However I don’t think this is feasible since we have to many VCS’s around.