`make-shell`, a modular replacement for `mkShell`

[nicknovitski]

Turn THIS:

pkgs.mkShell {
  packages = [pkgs.curl pkgs.git pkgs.jq pkgs.wget];
};

into THIS!

pkgs.make-shell {
  packages = [pkgs.curl pkgs.git pkgs.jq pkgs.wget];
};

Alright, I admit that this project won’t help many simple use cases, but changing mkShell to accept a module as its argument is helping me manage a herd of devShells, using imports to manage common features, and to define new options.

I wrote a full rationale for the project in a WHY.md document, which might tell you if you could get any use from it, but if you prefer to see that by example, here’s a “Shell Module” which adds and authenticates a configurable Google Cloud CLI package:

{
  lib,
  pkgs,
  config,
  ...
}: {
  options.gcloud.enable = lib.mkEnableOption "install google cloud sdk";
  options.gcloud.extraComponents =
    lib.mkOption
    {
      description = "IDs of Google Cloud SDK components to install (list them with `gcloud components list`)";
      default = [];
      example = ["gke-gcloud-auth-plugin"];
      type = lib.types.listOf lib.types.nonEmptyStr;
    };
  config = lib.mkIf config.gcloud.enable {
    packages = [
      (
        if config.gcloud.extraComponents == []
        then pkgs.google-cloud-sdk
        else
          pkgs.google-cloud-sdk.withExtraComponents
          (builtins.map (c: pkgs.google-cloud-sdk.components.${c})
            config.gcloud.extraComponents)
      )
    ]; 
    shellHook = ''
        gcloudAccount=$(gcloud auth list --filter=status:ACTIVE --format="value(account)")
        if [ -z ''${gcloudAccount:+set} ]; then
          echo "Google Cloud SDK is not authorized"
          read -rp "In the browser tab about to open, authenticate to your Google Cloud account. (press enter to continue)"
          gcloud auth login
        fi
        if ! gcloud auth application-default print-access-token&>/dev/null; then
          read -rp "You don't have any default application credentials for Google Cloud. In the browser tab about to open, authorize the use of your Google Cloud account. (press enter to continue)"
          gcloud auth application-default login
        fi
    '';
  };
}

…and here’s a simple option which uses that “gcloud” module to set a developer up to authenticate with clusters running in google cloud’s kubernetes engine:

{
  lib,
  pkgs,
  config,
  ...
}: {
    options.k8s.enable = lib.mkEnableOption "kubernetes tools";
    config = lib.mkIf config.k8s.enable {
        packages = [pkgs.kubectl pkgs.minikube];
        gcloud.enable = true;
        gcloud.extraComponents = ["gke-gcloud-auth-plugin"];
    };
};

Commit those files to a flake and then you could write a consuming flake like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
    make-shell.url = "github:nicknovitski/make-shell";
    my-org-nix.url = "github:myOrg/nix-config"
  };

  outputs = {
    self,
    nixpkgs,
    flake-utils,
    make-shell,
    my-org-nix,
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {
        inherit system;
        config = {};
        overlays = [make-shell.overlays.default];
      };
    in {
      devShells.gcloud-function = pkgs.make-shell {
        imports = [my-org-nix.shellModules.default];
        gcloud.enable = true;
      };
     devShells.kubernetes = pkgs.make-shell {
       imports = [my-org-nix.shellModules.default];
       k8s.enable = true;
     };
    });
}

It also has a flake.parts flake module, if you like those. I hope at least a few people find it useful!

[jjpe]

@nicknovitski

Turn THIS:

pkgs.mkshell {
  packages = [pkgs.curl pkgs.git pkgs.jq pkgs.wget];
};

into THIS!

pkgs.make-shell {
  packages = [pkgs.curl pkgs.git pkgs.jq pkgs.wget];
};

The argument to each of these fns is identical. As-is, I don’t see the point of pkgs.make-shell at all.

Could you come up with an example that actually demonstrates the virtues pkgs.make-shell is supposed to have?

[piegames]

Please read the rest of the post

[jjpe]

I did, and yet I wrote that post. What’s that tell ya?

[jchw]

Well, the rest of the post is an example of using it in a way that shows off what it’s useful for. It’s mkShell but modular, because the input is a NixOS module, that can include other modules and configuration/options.

[lbodor]

Thank you very much for sharing make-shell. It is exactly the lean alternative to existing options I’d been hoping someone would write.

[nicknovitski]

Haha! I’m not bothered if make-shell seems useless to you; as my joking first example was meant to show, in many situations, it is useless!

Also I do struggle to explain where it isn’t useless in a small number of words. In the project’s WHY, I try to describe the situations where I think people could get the most benefit from it, and highlight all of the contrasting situations where I think they wouldn’t. I wrote that document because I know there are many, many users of nix, and even of mkShell who have no strong reason to use make-shell, and I want them to be able to recognize that as soon as possible.

I linked the WHY document instead of copying it to my original post because it’s long, but I can summarize the primary experience that made me want to write make-shell: it was working across 10+ flakes in one organization, all with at least one devShell, in all of which developers expected a few common dependencies to be present, but which some developers didn’t want to install globally.

Here’s how to achieve that with make-shell:

# my-org.nix/flake.nix
https://flake.parts/options/make-shell
outputs: _ {
  shellModules.default = { pkgs, ...}: { packages = [pkgs.jq pkgs.yq pkgs.fd]; };
}
# 
# our-python-project/flake.nix
inputs.make-shell.url = "github:nicknovitski/make-shell";
inputs.my-org-nix = "path:../my-org.nix";
outputs = {
    nixpkgs,
    flake-utils,
    make-shell,
    my-org-nix,
    ...
  }:
    flake-utils.lib.eachDefaultSystem (system: let
      pkgs = import nixpkgs {
        inherit system;
        config = {};
        overlays = [make-shell.overlays.default];
      };
    in {
  devShells.default = pkgs.make-shell {
    imports = [my-org-nix.shellModules.default];
    packages = [ pkgs.python3 ]; # option definitions here are merged with the ones in the imported modules
  };
}

Is that a clearer use-case?

[nicknovitski]

Thank you so much for saying this, that’s exactly the use I hope people have from it. It’s true that make-shell is just a much, much worse version of devenv and devshell. It’s much less user-friendly, it has no CLI, no config file standard, almost no “features” at all! But if you’re repeating yourself in writing shell definitions for your flakes, you could use it to write as many of those missing features which you want and have time for. And maybe we can move towards open source collections of re-usable Shell Modules which could be re-used by those superior projects? I intend to write one of those next, to see if it’s a good idea or not.

Oh, and the flake module’s documentation has been published on flake.parts now.

[emily]

I like things that are much worse. Have you considered offering a variant that makes a “bare” shell without any of the usual stdenv stuff? Getting a direnv setup that doesn’t set 500 environment variables or override the macOS shell utilities with GNU coreutils is most of why I use devshell.

[nicknovitski]

I have written modules which make a little progress towards that, although I haven’t decided whether or where to publish them.

One of make-shell’s options is function, which is pkgs.mkShell by default, but can be replaced with any function which supports the same arguments. And one actual “feature” that make-shell has over mkShell is that the env option can contain attributes whose value is null, which causes the variable of that attribute’s name to be unset in the shell.

So a simple module which used both of those features to slim things down a bit could look like this:

{pkgs,...}: {
  function = pkgs.mkShellNoCC;
  env = { # credit to https://github.com/cachix/devenv/blob/7b3ed618571f0d14655b05f7b1c6ef600904383a/src/modules/top-level.nix#L79
    HOST_PATH = null;
    NIX_BUILD_CORES = null;
    __structuredAttrs = null;
    buildInputs = null;
    buildPhase = null;
    builder = null;
    depsBuildBuild = null;
    depsBuildBuildPropagated = null;
    depsBuildTarget = null;
    depsBuildTargetPropagated = null;
    depsHostHost = null;
    depsHostHostPropagated = null;
    depsTargetTarget = null;
    depsTargetTargetPropagated = null;
    dontAddDisableDepTrack = null;
    doCheck = null;
    doInstallCheck = null;
    nativeBuildInputs = null;
    out = null;
    outputs = null;
    patches = null;
    phases = null;
    preferLocalBuild = null;
    propagatedBuildInputs = null;
    propagatedNativeBuildInputs = null;
    shell = null;
    shellHook = null;
    stdenv = null;
    strictDeps = null;
  };
}

But I’m sure this could be improved a great deal, especially for function.

[emily]

Yeah, I think starting from scratch rather than trying to trim mkShell down to size makes more sense. devshell has mkNakedShell, but it’s not publicly‐exposed (at least in the flake).

[nicknovitski]

I’ve been subscribed to this PR for a while, it may show a good way forward, whether or not it’s right for nixpkgs: mkShellMinimal: Create an ultra minimal nix-shell

[fzakaria]

I always thought my PR for a minimal shell then adding layers (I guess it could be modules) could be a viable alternative.

You could build up the default mkShell back from it.

I wish I landed that PR you linked

[nicknovitski]

I thought it might also be helpful to have a fully worked shell module as an example, so I made this: GitHub - nicknovitski/javascript-nix: `make-shell` nix modules for javascript development.

Actually, I know it will be helpful…to me! But I hope it will be helpful to other people too, whether to see what’s possible with make-shell, or as the start of configuration for a project or two.

e: I already hope to extend it to have a version-selection capability in the vein of the very impressive nixpkgs-python.

[nicknovitski]

Oh, interesting. So it wouldn’t be impossible to maintain the mkShell interface while slimming down the implementation to something like your minimal shell…that seems promising!

[jjpe]

It definitely is, thank you for that explanation

[lucasew]

What is the differential for this compared to cachix devenv?

[nicknovitski]

As I said above, but I don’t mind saying again, the difference is it’s much worse! It’s a function you can call in your flakes which takes a module and returns a shell derivation, and that’s it. devenv is a command line tool which does not require you to work with flakes, has many great subcommands like search and up, includes I-don’t-know-how-many option declarations contributed and maintained by I-don’t-know-how-many people (including myself), and is supported by a great company. make-shell has zero high-level option declarations, and I wasn’t planning on adding any, except in other flakes like javascript-nix.

If you’re evaluating options for adopting nix for a project, my advice is you shouldn’t even consider make-shell. Maybe that’s something I should emphasize in the why file…

[nicknovitski]

I just released v1.2.0, which has a big not-breaking change: the deprecation of the function option and the addition of the stdenv option.

I think makes a little more sense; function always felt strange to me.

let
  thisEmitsAWarning = make-shell ({pkgs, ...}: { function = pkgs.mkShellNoCC; });
  useThisInstead = make-shell ({pkgs, ...}: { stdenv = pkgs.stdenvNoCC; });
in
  thisEmitsAWarning == useThisInstead

[nicknovitski]

I’ve released another shell module, GitHub - nicknovitski/gcloud-nix: A shell module for adding the Google Cloud SDK.

The Google Cloud SDK includes the gcloud command line tool, which has a components subcommand for installing a variety of other tools and packages. Some of those tools have separate packages in nixpkgs, but all of the components are also packaged as components in nixpkgs, as an interesting interface for the package google-cloud-sdk:

pkgs.google-cloud-sdk.withExtraComponents [
  pkgs.google-cloud-sdk.components.kubectl
  pkgs.google-cloud-sdk.components.gke-gcloud-auth-plugin
  pkgs.google-cloud-sdk.components.skaffold
]

gcloud-nix lets me make this terser, and merges component definitions, so I can do something like this:

# gke.nix
{pkgs, ...}: {
  gcloud.enable = true;
  gcloud.extra-components = [ "kubectl" "gke-gcloud-auth-plugin" "skaffold" ];
}

# For most projects:
pkgs.make-shell {
  imports = [./gke.nix];
  # project-specific options
}

# For a project which also uses pubsub:
pkgs.make-shell ({pkgs, ...}: {
  imports = [./gke.nix];
  gcloud.extra-components = [ "pubsub-emulator" ]; # concatenated cleanly with gke.nix!
  packages = [ pkgs.openjdk ]; # necessary for the emulator to run
  shellHook = "gcloud beta emulators pubsub env-init"; # set environment variables so pubsub applications run in this shell will target the emulator
})

Relatedly, I’m also considering upstreaming an option to make-shell which I’ve found use for in a few projects: interactiveShellHook. It basically does this:

config.shellHook = lib.mkAfter ''
    if [[ -n "''${_nix_direnv_force_reload:-}" || $- == *i* ]]; then
      ${config.interactiveShellHook}
    fi
'';

The conditional is true when someone is interactively running nix-shell or nix-develop, or nix-direnv-reload from nix-direnv’s manual mode (GitHub - nix-community/nix-direnv: A fast, persistent use_nix/use_flake implementation for direnv [maintainer=@Mic92 / @bbenne10]). Otherwise it is false.

mkAfter means that this will be added to the end of shellHook, which made sense to me: conditionally evaluated code coming after code which is always evaluated.

It can be used for things like this:

pkgs.make-shell {
  env.AWS_PROFILE = "project-profile";
  packages = [pkgs.awscli2];
  interactiveShellHook = ''
    if ! aws configure list; the
      echo "Set your AWS access key id and secret access key! (Other values can be blank)"
      aws configure --profile "$AWS_PROFILE"
    fi
  '';
}

pkgs.make-shell {
  gcloud.enable = true;
  gcloud.extra-components = ["gke-gcloud-auth-plugin" "kubectl"];
  interactiveShellHook = "gcloud container clusters get-credentials myClusterName";
}