Handling default.nix in projects that I develop myself

savannidgerinel · February 3, 2020, 11:40pm

What is the best practice for creating a nix derivation for a project that I am actually developing myself?

To clarify, I have built several derivations for other projects (sadly, all somewhat incomplete or not suitable for distribution), but the derivations for my own projects have proven tricky. Typically, I create a shell.nix file for every project, and a default.nix file for distributable executables. here is one example.

Do other developers do this, or do they just keep the Nix derivation files 100% within the nixpkgs repo?

A drawback that I may have found with my method today is that, despite my efforts to make a repeatable experience, something in 19.03 changed and my derivation is no longer being consistent about the Rust environment it uses to build things. Some crates are getting built with 1.40, others are getting built with 1.39, and I can’t help but feel that this problem will not exist, or be easier to manage, if I was distributing along with nixpkgs best practices.

danieldk · February 4, 2020, 7:05am

I use shell.nix in my projects and in some cases also provide a default.nix. However, I have found that things break sometimes, especially when people are using different nixpkgs versions. So, I have started to pin nixpkgs and other dependencies with niv. For example:

github.com

danieldk/sentencepiece/blob/dbb4e387c1b35c445bbccd5bc1545c0a65c3dcf3/shell.nix

# We pin nixpkgs to improve reproducability. We don't pin Rust to a
# specific version, but use the latest stable release.

let
  sources = import ./nix/sources.nix;
  nixpkgs = import sources.nixpkgs {};
  danieldk = nixpkgs.callPackage sources.danieldk {};
  mozilla = nixpkgs.callPackage "${sources.mozilla}/package-set.nix" {};
in with nixpkgs; mkShell {
  nativeBuildInputs = with nixpkgs; [
    clang
    mozilla.latest.rustChannels.stable.rust
    pkgconfig
    protobuf
    rust-bindgen
  ];

  buildInputs = with nixpkgs; [
    sentencepiece
  ];

This file has been truncated. show original

This gives full control over what build environment is used (in this example I don’t pin Rust, because I like to compile with the latest stable).

savannidgerinel · February 4, 2020, 1:08pm

Maybe I can use that to pin some of the extra Rust build infrastructure. I haven’t had time to investigate (and nixpkgs investigations are very, very difficult for me), but I do feel like the problem lay in a behavior change in buildRustCrates or some such.

tomberek · February 4, 2020, 2:16pm

Flakes are a new development to help projects structure themselves.

savannidgerinel · February 4, 2020, 2:17pm

Yes, but they are still under development, and pretty mysterious to me.

savannidgerinel · February 12, 2020, 2:39pm

These answers didn’t feel quite complete, so I asked in #irc about it. The answer from them is that nixpkgs doesn’t allow us to run a derivation that is fetched from GitHub. I didn’t dig deeply into the reasons for it, but that means that the solution is to create the derivation directly in nixpkgs.

Further, crates-io.nix and Cargo.nix are large enough files that they prefer to not have those in the repo, either. So, I talked with them some and then restructured my project slightly so that I could just use buildRustPackage to build my program. I’ll add it to nixpkgs as soon as I do my next release.

cdepillabout · February 13, 2020, 4:36am

This feature is known as “import-from-derivation (IFD)”.

This isn’t allowed in nixpkgs due to a limitation(?) of Hydra.

Here’s a nice resource explaining IFD: