Flakes: cognitive overload, gateway drugs, self-contained simple recipes

Having read all the material I could find on the topic of flakes, I find that I cannot discern (in a reasonable amount of time) between what is essential to the functioning of some flake-based solution and what is over-engineering or irrelevant to my needs or one person’s vision of how something might be implemented or …

Many of the examples I have found, seem to attempt to provide complete personal-devops solutions. While this is, undoubtedly, a very valuable and compelling use-case, I don’t think it helps at all in trying to persuade Joe User that Nix is something relevant and immediately valuable. I’m after gateway drugs into Nix, ones that are affordable for those who have not yet fully bought into Nix, and even those who have never heard of Nix before.

In this context I would like to solicit your help in coming up with some self-contained flake-based solutions to a couple of relatively simple problems of limited scope, the hope being that these might

  • give me some insight into the essential elements of flake-based solutions and a foundation on which I will be able to build more complex structures of my own;

  • be shown to non-Nix-users as tractable recipes that they can realistically apply in their own environments, rather than complex horror stories that will make them run screaming away from Nix.

I also suspect that there are others in an a similar situation to mine, who fall in the gap between the very sparse flakes documentation, and the overly complex examples one can find on GitHub and elsewhere; who might also benefit from seeing simpler, more focused solutions.

Problem 1: Developer and user environments for a software project

I am used to writing a shell.nix which declares all the dependencies required to build my project, and using this with direnv to automatically enable (complete, self-contained, isolated) development environments for a bazillion different projects that I’m working on.

I am aware that, with some fiddling, one can write a derivation.nix suitable for inclusion in nixpgks, which can be used from the shell.nix

What is the simplest flake-based structure which needs to be built around a nixpkgs-compatible derivation.nix, that would make

  • nix run
  • nix develop
  • nix shell
  • nix build
  • nix flake check

work sensibly?

Prerequisites

  1. Nix >= 2.4(?) is enabled in the user’s environment

Problem 2: Bootstrapping home-manager

Home manager is wonderful. It brings the benefits of declarative, reproducible and portable configurations to managing personal environments. I want to give my colleagues the gift of home-manager.

[Aside: It irks me that most introductory Nix material encourages the use of nix-env -i. Yes, it’s is much better than apt, yum, pacman, etc. in many ways, but it fails to take advantage of some of the greatest benefits of Nix by being imperative rather than declarative: it misses the point!]

To this end I want it to be possible to say

nix --experimental-features = 'nix-command flakes' run github:username/my-config-files

with the following prerequisites and results.

Prerequisites

  1. Nix >= 2.4(?) is installed on the machine
  2. Somehow nix must have been enabled in the user’s account. Not sure what this would look like for a user who has never used Nix before, on a non-NixOS machine with a pre-existing multi-user install.

Results

  1. A clone of github:username/my-config-files is present in the home directory of the user who ran the command.
  2. home-manager has activated the environment declared in my-config-files.
  3. home-manager switch works, using the environment declared in my-config-files.
  4. The new nix CLI should work for this user without the need of --experimental-features = 'nix-command flakes'.

Hopes

This single command gives someone who is new to Nix

  1. a working home-manager installation.
  2. a simple route for reusing this environment on different machines, by making a personal clone of my-config-files.

I hope that the bang-for-the-buck ratio will be a compelling reason to start using Nix.

Problems

Enabling flakes on NixOS vs non-NixOS seems to require completely different mechanisms. Can a one-fits-all solution be provided?

More generally speaking, I’m experiencing problems and inconsistencies with getting flakes to work reliably: e.g.:

vagrant@debian10:~$ nix --experimental-features 'nix-command flakes' flake new flake-new
warning: ignoring the user-specified setting 'experimental-features', because it is a restricted setting and you are not a trusted user

… but now I see that, in spite of the warning to the contrary, the command appears to succeed. [This spurious warning has wasted at least 30 minutes of my life today, and probably a lot more in the past: I recall banging my head against it on other occasions, but only now realize that it is spurious!]

Obvious enhancements

If the environment is to be reusable, it should be easily adaptable to different types of machines: personal laptop, development machine, production machine, etc. Therefore it would be useful to split the environment declaration into composable components which describe what is needed in different roles.

Start simple

While modularisation is valuable, I think that the initial example needs to be simple and un-overwhelming, so I’d like to start of with something as simple as:

{ config
, pkgs
, ... }:

let
  link = config.lib.file.mkOutOfStoreSymlink;
in
{
  home.file.".something-read-only" .source =      ../something-read-only;
  home.file.".something-read-write".source = link ../something-read-write; # Note 1

  home.packages = with pkgs;
    [
      home-manager # Note 2
      cowsay
    ];

  # Note 3
  programs.direnv.enable = true;
  programs.direnv.enableNixDirenvIntegration = true;
}

Notes

  1. Having to re-run (the all-too-often excruciatingly slow home-manager switch) on every experimental tweak of some utility’s configuration file, is hugely painful. I consider the ability to get immediate feedback on config file changes to be crucial. This link alias for mkOutOfStoreSymlink allows for easy switching into ‘experimental mode’ and thus makes the experience tolerable.

  2. In my (horrible, convoluted) nix-shell-based approach to bootstrapping home-manager, I had home-manager install itself. Maybe this isn’t the best way to go.

  3. direnv + nix-shell/nix develop is a killer feature!

4 Likes

I think you don’t necessarily need the derivation.nix, but can just use buildEnv.
buildEnv doesn’t accept a shell hook, so you’d have to add it to the mkShell args then.

Assuming “suitable for inclusion in nixpkgs means something like

{ stdenv, <deps> }: stdenv.mkDerivation { ... buildInputs = [ <deps> ]; }

or

{ buildEnv, <deps> }: buildEnv { ..., paths = [ <deps> ]; }`

so you can use callPackage, the flake could probably look like this:

{

  # optional: Nix infers 'nixpkgs', not sure which branch
  inputs.nixpkgs.url = "github:NixOS/nixpkgs";

  outputs = { self, nixpkgs }:
  let
    system = "x86_64-linux";
    pkgs = nixpkgs.legacyPackages.${system};
  in
  {
    # used by
    #   nix run
    #   nix shell
    #   nix build
    defaultPackage.${system} = pkgs.callPackage ./derivation.nix {};

    # used by
    #   nix develop
    devShell.${system} = pkgs.mkShell {
      # use 'buildInputs' if using 'devEnv'
      inputsFrom = [ self.defaultPackage.${system} ];
    };
    # or devShell.${system} = import ./shell.nix { inherit pkgs; };
    # if the shell.nix is in equivalent to the mkShell call here.

    # used by 
    #   nix flake check
    checks.${system}."<attr>" = <some check derivation; test is derivation build>

    # extras:
    # used by 
    #   nix run
    defaultApp.${system} = { type = "app"; program = "<path to the executable; are commandline args allowed?>"; };
  };
}

To make it more flexible, you can e.g.

  • define the dev env in packages."${system}".devEnv = ...
    and set defaultPackage."${system}" = self.packages."${system}".devEnv
  • use a list of systems and map over them (like eachSystem from "github:numtide/flake-utils").

Direnv integration

Either refer to the wiki page or use something like this .envrc:

mkdir -p "$(direnv_layout_dir)"
watch_file *.nix
eval "$(nix print-dev-env --profile "$(direnv_layout_dir)/flake-profile" .)"
1 Like

Did you mean something like pkgs = import nixpkgs { inherit system; };? (Or have I screwed something up at my end?)

In fact, I made a mistake there (now fixed in above post): It must be nixpkgs.legacyPackages.${system}.
You only need to use the nixpkgs specific api (use nixpkgs as a function) if you want to pass additional arguments like config or overlays.

I’m now banging my head against the rather unhelpful

error: --- SysError ------------------------------------------------------------------------------------------------------------ nix
getting status of '/nix/store/mrcp47gx7krl1gmrkqr743i394rh7n5f-source/nix': No such file or directory

failure of both nix run and nix shell.

If no branch is specified, then the “default” branch will be used. In case of the mentioned repository, the default branch is “master”.

I meant, I’m not sure which branch is chosen if you omit the input declaration but only list nixpkgs as an arg to outputs.

Then its resolved against the registry.

You can take a look at it using nix registry list.

And per default flake:nixpkgs does indeed resolve to github:nixos/nixpkgs. One of those defaults I am not very happy with…

Can you show us your setup? My flavor of the flake works.

Are you in a git repository or something? Nix will only consider tracked files so you’ll need to git add them.

See: Be smart and helpful when some files are missing during the evaluation · Issue #4507 · NixOS/nix · GitHub

Bingo! I hadn’t staged derivation.nix.

How would you recommend generalizing the hard-wired system = "x86_64-linux"?

system is always explicit. So you can use e.g.

let systems = [...]; in nixpkgs.lib.forEach systems (system: {
  defaultPackage.${system} = ...
  ...
}

but then you must know which systems it is compatible with.
You can also take the more convenient path and add inputs.flake-utils.url = github:numtide/flake-utils
and use then in the outputs

outputs = { self, nixpkgs, flake-utils}:
  flake-utils.eachDefaultSystem (system:
  let pkgs = nixpkgs.legacyPackages.${system}; in {
    defaultPackage = pkgs.callPackage ...
    ...
  });

or

outputs = { self, nixpkgs, flake-utils}:
  with flake-utils; eachSystem allSystems (system: 
  let pkgs = nixpkgs.legacyPackages.${system}; in {
    defaultPackage = pkgs.callPackage ...
  ...
  });

See the README of GitHub - numtide/flake-utils: Pure Nix flake utility functions [maintainer=@zimbatm].

I think you’ve missed out a lib in the middle of flake-utils.eachDefaultSystem

I asked because I wasn’t having much luck with flake-utils. This version with a hard-wired system works for me:

{
  description = "A simple example of managing a project with a flake";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-20.09";
  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = nixpkgs.legacyPackages.${system};
    in
      {
        defaultPackage.${system} = pkgs.callPackage ./nix/derivation.nix {};
      };
}

but this attempt to generalize the system by using flake-utils fails:

{
  description = "A simple example of managing a project with a flake";
  inputs.nixpkgs    .url = "github:NixOS/nixpkgs/nixos-20.09";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system}; in
        {
          defaultPackage.${system} = pkgs.callPackage ./nix/derivation.nix {};
        }
    );
}

with the error

flake output attribute 'defaultPackage.x86_64-linux' is not a derivation

And I don’t see how to reconcile your outline above with what I find in the README or with what I already have.

You’re right that there must be a lib. So the correct expression is

{
...
  outputs = { self, nixpkgs, flake-utils}:
    flake-utils.lib.eachDefaultSystem (system:
    let pkgs = nixpkgs.legacyPackages.${system}; in {
      defaultPackage = pkgs.callPackage ...
      ...
    });
}

Look carefully: It’s defaultPackage = ..., not defaultPackage.${system} = .... You can also see this in the example for eachSystem in the README of flake-utils.

Right, I think I’ve finally figured it out. Thank you!

[Aside:

What threw me for a loop was

  1. I thought that your defaultPackages (note the s at the end) was some piece of magic whose documentation/definition I couldn’t find; but now I’m pretty sure that it’s just a typo for defaultPackage.

  2. pkgs in your example is undefined, and I was trying to understand how the (non-existent) magic defaultPackages (with trailing s) somehow made it appear.

]

I now have

{
  description = "A simple example of managing a project with a flake";
  inputs.nixpkgs    .url = "github:NixOS/nixpkgs/nixos-20.09";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system}; in
      {
        defaultPackage = pkgs.callPackage ./nix/derivation.nix {};
      }
    );
}

which seems to be working.

Thanks again!

Ah sorry! I fixed both above. One typo, copied.

TLDR

Is it possible to automatically access the appropriate .${system} of packages other than the implicit defaultPackage?

Details

I have this code:

outputs = { self, nixpkgs, flake-utils }:
  flake-utils.lib.eachDefaultSystem (system:
    let pkgs = import nixpkgs { inherit system; }; in
    {
      defaultPackage = pkgs.stdenv.mkDerivation {
        name = "whatever";
        # THE DERIVATION BODY
      };

      defaultPackage-but-with-different-attribute-name = pkgs.stdenv.mkDerivation {
        name = "whatever";
        # *IDENTICAL* COPYPASTA of THE DERIVATION BODY above
      };
    });

that is to say: two identical derivations, differing only in the names to which they are bound in the outputs set.

nix flake show gives

├───defaultPackage
│   ├───aarch64-linux: package 'whatever'
│   ├───i686-linux: package 'whatever'
│   ├───x86_64-darwin: package 'whatever'
│   └───x86_64-linux: package 'whatever'
└───defaultPackage-but-with-different-attribute-name: unknown

but this hides the fact that defaultPackage-but-with-different-attribute-name has the same internal structure as defaultPackage:

  1. nix build works

  2. The following fail with equivalent error messages:

    flake output attribute 'defaultPackage(-etc) is not a derivation
    
    • nix build .#defaultPackage
    • nix build .#defaultPackage-but-with-different-attribute-name
  3. The following succeed

    • nix build .#defaultPackage.x86_64-linux
    • nix build .#defaultPackage-but-with-different-attribute-name.x86_64-linux

My questions:

  1. Why is the internal structure of the non-defaultPackages hidden by nix flake show?
  2. Is it possible to access the appropriate .${system} of non-defaultPackages automatically, just like nix build does for the implicit defaultPackage?

It is not hidden, though defaultPackage-but-with-different-attribute-name is none of the “well-known” top-level outputs and therefore nix flake show and nix flake check do not know how to deal with them.

You can read more about the “well-known” outputs and how they have to look like to be recognized by nix flake show in the wiki about flakes.

Move it to packages.${system}

and use nix (build|run|shell|...) <flake>#<package> for packages.${system}.<package>.