Is this a sensible way to setup nix-shell for development and CI?

Help
1

I’m taking my first major steps with nix and I’m wondering how people are setting up their nix-shell development environments.

Is this a sensible way to be doing things?

{
  stable ? import (builtins.fetchTarball {
             url = "https://github.com/NixOS/nixpkgs/archive/18.09.tar.gz";
             # Hash obtained using `nix-prefetch-url --unpack <url>`
             sha256 = "1ib96has10v5nr6bzf7v8kw7yzww8zanxgw2qi1ll1sbv6kj6zpd";
           }) {},
  unstable ? import (builtins.fetchTarball {
               url = https://github.com/nixos/nixpkgs-channels/archive/44b02b52ea6a49674f124f50009299f192ed78bb.tar.gz;
               # Hash obtained using `nix-prefetch-url --unpack <url>`
               sha256 = "0gmk6w1lnp6kjf26ak8jzj0h2qrnk7bin54gq68w1ky2pdijnc44";
             }) {}
}:
with stable;
stdenv.mkDerivation {
  name = "env";
  buildInputs = [
    awscli
    unstable.bazel
    git
    kubectl
    python37
    ...
  ];
}

What I’m looking for here is creating a reproducible environment in a declarative way, with dependencies, but also a way to pull in certain packages from unstable. It appears to work, but I’m not sure if it’s a terrible idea.

The next step on top of that, would be adding custom derivations for things that are not yet in nix. Or I guess I can just create a repo for custom derivations and then have (stable, unstable, custom).

Thoughts?

2

This will work but you now need to track multiple nixpkgs. Also, you’ll likely run into problems mixing python dependencies from stable and unstable. I like to use overlays for this and separate out the package definition. I typically have a

  • package.nix containing {stdenv, awscli, git, python, ...}: stdenv.mkDerivation...
  • release.nix which imports the appropriate nixpkgs, applies overlays if any, then callPackages the package.nix

Does this fit your use case
Nixpkgs 23.11 manual | Nix & NixOS ?

Edit: fix link, typo

1 Like
3

If you don’t care for using Nix for the actual build but only want to get a shell with dependencies, there’s also mkShell: Nixpkgs 23.11 manual | Nix & NixOS

1 Like
4

Thanks. I’ll check it out.

My actual use-case is to use Bazel to build components inside a monorepo, but I’d like to use Nix for managing the handful of external toolchains that Bazel doesn’t easily provide.

i.e. I’d like to pin Python and other utility binaries that we use for our local developement, CI test and build.

So, essentially, I imagine that developers and CI could enter the nix-shell with --pure and should be able to execute tests and build artifacts.

5

Yes, sounds sensible to me, including an expression like the one above.

6

I’ve recently refined this and I’d like to share with others to get feedback.

Contents of overlays.nix

self: pkgs:

let
  bazel = pkgs.callPackage ./bazel/default.nix {};
  bazel-buildtools = pkgs.callPackage ./bazelbuildtools/default.nix {};
in
{
  paths = [
    bazel
    bazel-buildtools
  ];
}

Contents of overlays/bazel/default.nix

self: super:
let
  unstable = import (
    fetchTarball {
      url = https://github.com/nixos/nixpkgs-channels/archive/44b02b52ea6a49674f124f50009299f192ed78bb.tar.gz;
      # Hash obtained using `nix-prefetch-url --unpack <url>`
      sha256 = "0gmk6w1lnp6kjf26ak8jzj0h2qrnk7bin54gq68w1ky2pdijnc44";
    }
  ) {};
in {
  bazel = unstable.bazel;
}

Contents of custompkgs folder

The custompkgs folder has subfolders where Nix expressions for custom built software which isn’t available on nixpkgs yet (or are in a branch elsewhere on github) are stored.

Contents of default.nix

with import (
  fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/18.09.tar.gz";
    # Hash obtained using `nix-prefetch-url --unpack <url>`
    sha256 = "1ib96has10v5nr6bzf7v8kw7yzww8zanxgw2qi1ll1sbv6kj6zpd";
  }
) { overlays = [ (import ./overlays/overlays.nix) ]; };
let
  custom1 = pkgs.callPackage ./custompkgs/custom1/default.nix {};
  custom2 = pkgs.callPackage ./custompkgs/custom2/default.nix {};
in
stdenv.mkDerivation rec {
  buildInputs = [
    bazel
    bazel-buildtools
    custom1
    custom2
    go
    kubectl
    libffi
    openssl
    python37
    python37Packages.pip
  ];
  shellHook = ''
    # unset SOURCE_DATE_EPOCH so that we can use python wheels
    # https://github.com/NixOS/nixpkgs/blob/5094de98f0998980d20efa1bce488ee89e922224/doc/languages-frameworks/python.section.md#python-setuppy-bdist_wheel-cannot-create-whl
    # This avoids "ValueError: ZIP does not support timestamps before 1980"
    # Nix sets file mtime to 1, but Python Zip format expects DOS time starting at 1980.
    unset SOURCE_DATE_EPOCH
  '';
}

This appears to do what I intend.

One thing that I’m not sure of here, is what is the difference between the overlays vs custom packages?

To me, it seems like overlay only allowed to overlay a package that already exists with a different version or different configuration?

If I wish to combine existing Nix packages with custom Nix packages, I have to explicitly call the expressions.

Is there a way to load those custom packages in a single import or “callPackages” so that in the default.nix I can just concat 2 lists. One of standard nixpkgs and one of custom nixpkgs.

7

Hmmm… actually, no. The overlays above are not actually working. The packages being used are from nix stable, not from unstable, even though I tried to override them with overlays.