Creating a nix shell (default.nix) with an overlay for development environments

Guides
1

Marked Howto in the hope some best practices fall out of this thread.

I am creating a default.nix file for an open source library. Contributors should be able to clone the project, run nix-shell, and be ready to contribute, build documentation, and so on. So long as I stay within the bounds of the current Nix package set this is straightforward to create. But there is one package which needs a more recent version than is in the 18.09 channel. I’d like to update this single package, only in this default.nix file (not system-wide). That sounds like a job for overlays, but I’m having trouble getting this to work.

Here’s the default.nix file without the package that needs to be updated. It works fine:

{ nixpkgs ? import <nixpkgs> {  } }:
 
let
  pkgs = [
    nixpkgs.nodejs
    nixpkgs.yarn
    nixpkgs.stack
  ];
 
in
  nixpkgs.stdenv.mkDerivation {
    name = "env";
    buildInputs = pkgs;
  }

Now, I need to also add the mkdocs package in order to build the project documentation. The version available in nixpkgs is v0.17, but this project requires v1.0. So I need to adjust this default.nix to use a newer version of mkdocs than is available in the package set, and that sounds like a job for an overlay.

Fortunately, there is already a Nix file describing mkdocs 1.0.4 in a newer package set:

My plan was to use this expression to create an overlay in the default.nix file. I copy/pasted the expression into the default.nix file as the variable mkdocs-1_0_4, then used that in an overlay to change the build environment to point to this package. (I’m not totally sure this is the proper way to go about this).

let
  mkdocs-1_0_4 =
    { lib, python, fetchFromGitHub }:
 
    with python.pkgs;
 
    buildPythonApplication rec {
      pname = "mkdocs";
      version = "1.0.4";
 
      src = fetchFromGitHub {
        owner = "mkdocs";
        repo = "mkdocs";
        rev = version;
        sha256 = "1x35vgiskgz4wwrvi4m1mri5wlphf15p90fr3rxsy5bf19v3s9hs";
      };
 
      checkInputs = [
        nose nose-exclude mock
      ];
 
      NOSE_EXCLUDE_TESTS = lib.concatStringsSep ";" [
        "mkdocs.tests.gh_deploy_tests.TestGitHubDeploy"
        "mkdocs.tests.config.config_tests.ConfigTests"
        "mkdocs.tests.config.config_options_tests.DirTest"
      ];
 
      checkPhase = "nosetests mkdocs";
 
      propagatedBuildInputs = [
        tornado
        livereload
        click
        pyyaml
        markdown
        jinja2
        backports_tempfile
      ];
 
      meta = {
        homepage = http://mkdocs.org/;
        description = "Project documentation with Markdown";
        license = lib.licenses.bsd2;
      };
    };
 
  overlay = self: super: {
    mkdocs = super.buildEnv {
      name = "mkdocs";
      paths = [ mkdocs-1_0_4 ];
    };
  };
 
in { nixpkgs ? import <nixpkgs> { overlays = [ overlay ]; } }:
 
let
  pkgs = [
    nixpkgs.nodejs
    nixpkgs.yarn
    nixpkgs.stack
    nixpkgs.mkdocs
  ];
 
in
  nixpkgs.stdenv.mkDerivation {
    name = "env";
    buildInputs = pkgs;
  }

Unfortunately this results in an error:

error: while evaluating the attribute 'buildInputs' of the derivation 'env' at /nix/var/nix/profiles/per-user/root/channels/nixos/pkgs/stdenv/generic/make-derivation.nix:177:11:

  while evaluating the attribute 'passAsFile' of the derivation 'mkdocs' at /nix/var/nix/profiles/per-user/root/channels/nixos/pkgs/stdenv/generic/make-derivation.nix:177:11:

    cannot convert a function to JSON

I spent some time Googling around for this error but came up short. Thinking on failure, a few things come to mind:

  1. Perhaps this expression can’t just be copy/pasted in like this
  2. Perhaps I have made a mistake when creating the overlay
  3. Perhaps the dependencies of this package also need their own overlays to move to the appropriate versions (I’d expect a different error in that case)

I posted here in the hopes of

  1. Figuring out this issue and getting a working default.nix for this project
  2. Learning best practices for creating this sort of file in the future

With regards to #2, I’d love to hear about how others approach this issue. Thanks in advance!

1 Like
2

Unless I would need to do something custom with the package (which is not the case if I understand you correctly?) my approach would be to bring in an individual package from another nixpkgs closure / channel. Occasionally that might cause issues but for the most part it works okay. Example:

{ nixpkgs ? import <nixpkgs> {  } }:

let channels = rec {
  pkgs-master = import (fetchTarball {
      url = "https://github.com/NixOs/nixpkgs/archive/master.tar.gz";
  }) {};
};
in with channels;

let
  pkgs = [
    nixpkgs.nodejs
    nixpkgs.yarn
    nixpkgs.stack
    pkgs-master.mkdocs
  ];

in
  nixpkgs.stdenv.mkDerivation {
    name = "env";
    buildInputs = pkgs;
  }

You could also bring in that package from a a local closure:

  pkgs-dev = import /path/to/your/nixpkgs {};

Or use any other means that nix has for bringing in e.g. specific revision of the object or whatnot.

Hope this helps and it’s not too late! :slight_smile: