Why don't nix hashes use base-16?

[Ericson2314]

It’s implemented, I just need to convince people to merge it (and probably fix some conflicts in the meantime).

[jonringer]

@volth

Just curious: what was the reason to make sri format default in the next Nix ?

I have no idea. I have just observed this trend when first reviewing PRs, then later when I made the jump to nixUnstable for flake support.

[jonringer]

@danieldk

Some fetchers, such as fetchCargoTarball abuse FODs though. FOD builds have network access. fetchCargoTarball uses that to do vendor dependencies with cargo . This is problematic, because the output becomes dependent on how Cargo (and its dependencies) behave. It is not a proper FOD, because the definition of a FOD is that its output is independent of anything else.

I view FODs as more “defined as their own content”; I would say this is okay as long as it’s repeatable. With a Cargo.lock, you have a good chance of being able to pull down the same content. You’re also not dependent on how cargo resolves dependencies, as resolution was done during the lock creation.

I have also ran into the issue of needing special ssh access to some resources, and really the only way to do this is with FODs which can use your user credentials.

Also, one of the issues with naersk (a buildRustPackage alternative) is that it doesn’t fully emulate “cargo behavior” because it can’t recursively pull git+ssh sources as noted in this issue. However, I guess you could argue that fetchCargoTarball would also be subject to change if the git+ssh urI isn’t stable.

[danieldk]

@jonringer

I view FODs as more “defined as their own content”; I would say this is okay as long as it’s repeatable. With a Cargo.lock, you have a good chance of being able to pull down the same content. You’re also not dependent on how cargo resolves dependencies, as resolution was done during the lock creation.

I understand your point of view, but all hashes have been broken in the past because of changes in cargo vendor, and that should not happen for a fixed-output derivation. Fetching of individual crate archives could be FODs, because they never change and we even have usable hashes in the lock files (you could fetchurl them). Setting up a vendored output path, especially when done by a program like cargo vendor (which can change) should IMO be a non-FOD derivation.

It would be possible to build crates without violating FODs. AFAIK fromTOML was added for such cases. We could use fromTOML to read a lock file, fetchurl or fetchcrate the dependencies (with the provided hashes) and then set up the vendoring using a regular derivation. This also avoids the issue where all cargoSha256 need to be fixed everywhere if some aspect of the vendoring changes. Unfortunately, for nixpkgs it would also mean that would we have to add the Cargo.lock files for individual packages (since we can’t use IFD).

I think buildRustPackage as it is, is the best we have now. I prefer the philosophy of buildRustCrate, but since it’s a our own reimplementation of Cargo in Nix + shell, it does not cover all edge cases. Also, it is not clear if adding large amounts of automatically generated Nix expressions to nixpkgs is the way to go. buildRustPackage abuses the notion of FODs, but I don’t think we have anything better if we don’t want to add lock files to nixpkgs.

[danieldk]

@volth

@jonringer

For unstable (2.4pre), it will be a 64bit encoded sha256 sri hash

Just curious: what was the reason to make sri format default in the next Nix ?
[+] more succinct (questionable, assuming constant sha256- prefix and = suffix)
[+] compatible with HTML’s identity attribute (but what for?)
[-] never published on websites next to tarballs, there are normally base-16
[-] contains + and / and thus not form a bound word it the eyes on text

[+] makes it easier to switch to other default hash algorithms in the future.

[jonringer]

@danieldk

I understand your point of view, but all hashes have been broken in the past because of changes in cargo vendor , and that should not happen for a fixed-output derivation.

You’re right. But it’s rare for it to cause issues.

The same could be said about fetchpatch where patchutils has to be pinned because it affects how the patches are normalized. So, use of fetchpatch is also not a “true FOD” in this regard either.

I’m also not a big fan of making FODs super granular as in the case of node packages. Doing PRs which include node changes are painful, as you have to make changes to the list, and then wait a very long time for the node-packages.nix to be generated. If another PR which affects the same file gets merged, then your PRs will get merge conflicts, and then you have to regenerate the file for the “chance” for it to be merged. Not to mention that these “pinned lists” take up a significant amount of space within the repository:

$ find ~/.nix-defexpr/channels/nixos/ -type f -exec du -h {} \; | sort -rh | head -10
11M	/home/jon//.nix-defexpr/channels/nixos/pkgs/development/haskell-modules/hackage-packages.nix
3.8M	/home/jon//.nix-defexpr/channels/nixos/programs.sqlite
3.8M	/home/jon//.nix-defexpr/channels/nixos/pkgs/development/node-packages/node-packages.nix
2.6M	/home/jon//.nix-defexpr/channels/nixos/pkgs/development/r-modules/cran-packages.nix
2.6M	/home/jon//.nix-defexpr/channels/nixos/pkgs/applications/editors/emacs-modes/recipes-archive-melpa.json
1.9M	/home/jon//.nix-defexpr/channels/nixos/pkgs/tools/typesetting/tex/texlive/pkgs.nix
980K	/home/jon//.nix-defexpr/channels/nixos/pkgs/top-level/all-packages.nix
816K	/home/jon//.nix-defexpr/channels/nixos/pkgs/top-level/perl-packages.nix
556K	/home/jon//.nix-defexpr/channels/nixos/pkgs/development/compilers/elm/packages/node-packages.nix
528K	/home/jon//.nix-defexpr/channels/nixos/pkgs/applications/version-management/gitlab/yarnPkgs.nix

Which is part of the significant bloat in nixpkgs:

[10:59:21] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ du -hcd0 ./* | tail -1
214M	total
[10:59:25] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ git co HEAD~50000 # Around Nov 2018
Note: switching to 'HEAD~50000'.
...
[11:02:07] jon@jon-desktop /home/jon/projects/nixpkgs ((058a96dc0e4...))
$ du -hcd0 ./* | tail -1
151M	total

I guess you could argue that the lines-of-code per package is low in these generated files. However, it does come at a significant space cost.

Maybe in the future, we could have package ecosystems just defined through flakes in nixpkgs, and curation of those ecosystems can be done in other repositories. It would be harder to test “how does this python code change affect the documentation generation of this haskell package”, but it would have some other benefits. For example, vim and emacs packages could be automatically updated, tested, and merged. Then updating the packages on nixpkgs would just be updating the flake.lock.

{
  inputs.vimPlugins.url = "github:nixos/vim-plugins";
  inputs.vimPlugins.nixpkgs.follows = self; # not sure if this would work

  ...

However, I do really like that nixpkgs is a monolith. Being able to query how your changes affects every possible package is a huge strength from a testing perspective.

[danieldk]

@danieldk

Fetching of individual crate archives could be FODs, because they never change and we even have usable hashes in the lock files (you could fetchurl them). Setting up a vendored output path, especially when done by a program like cargo vendor (which can change) should IMO be a non-FOD derivation.

It would be possible to build crates without violating FODs. AFAIK fromTOML was added for such cases. We could use fromTOML to read a lock file, fetchurl or fetchcrate the dependencies (with the provided hashes) and then set up the vendoring using a regular derivation. This also avoids the issue where all cargoSha256 need to be fixed everywhere if some aspect of the vendoring changes.

In case anyone wants to try, I have made a PR which implements this based on @edolstra’s import-cargo. While it may not be fit for nixpkgs, since it requires that Cargo.lock is available, you can use it to apply buildRustPackage in your own Rust projects without needing to specify cargoSha256.

Example from the unit tests:

{ rustPlatform }:

rustPlatform.buildRustPackage {
  pname = "basic";
  version = "0.1.0";

  src = ./.;

  cargoLock = {
    lockFile = ./Cargo.lock;
  };

  doInstallCheck = true;

  installCheckPhase = ''
    $out/bin/basic
  '';
}

[asymmetric]

@danieldk

Some fetchers, such as fetchCargoTarball abuse FODs though. FOD builds have network access. fetchCargoTarball uses that to do vendor dependencies with cargo . This is problematic, because the output becomes dependent on how Cargo (and its dependencies) behave. It is not a proper FOD, because the definition of a FOD is that its output is independent of anything else.

Linking this issue, which seems relevant.