Nix shells - tips, tricks, and best practices

toraritte · October 9, 2022, 1:26am

WARNING
None of this is authoritative! Please take everything with a grain of salt, and as this is a wiki post, please consider editing it. The whole point of this post is to initiate a discourse and to improve the official documentation; see the diagram in How to contribute to documentation.
Thank you!

1. About Nix shells
- 0.1 What are Nix shells?
- 0.2 Use cases (and sources of confusion)
  - 0.2.1 Develop Nix packages
  - 0.2.2 Create ad hoc environments
  - 0.2.3 Examples
    - 0.2.3.1 nix-shell -p
    - 0.2.3.2 nix-shell with a default.nix
- 0.3 Benefits of using Nix shells
- 0.4 How to start applications from the terminal that won’t close with the terminal?
1. One-liners with pinning
- 1.1 Get a single package with nix-shell -p + callPackage
- 1.2 Improvise a shell.nix on the terminal
  - 1.2.1 mkShell vs mkDerivation
  - 1.2.2 Advantages of mkShell
- 1.3 Nix shell using multiple packages with specific versions (i.e., multiple Nixpkgs revisions)
1. Using nix-shell with shell.nix
- 2.1 What is shell.nix?
- 2.2 Issues & solutions
  - 2.2.1 nix-shell messes with locales
  - 2.2.2 How to make nix-shell clean up after itself?
2.x Examples

TODO: Still a draft; have about 40 tabs open with nix-shell related stuff.

0. About Nix shells

0.1 What are Nix shells?

nix-shell and nix develop start an interactive Bash shell, and, depending on their arguments, they will

pre-populate the subshell with stuff (e.g., environment variables are set, applications will be available in PATH)
start and configure services (e.g., launch a PostgreSQL instance and set it up)
?

QUESTION: Does nix run belong here? What else is missing?

TODO: Weave the Development environment with nix-shell nixos.wiki article into this - or vice versa.

0.2 Use cases (and sources of confusion)

0.2.1 Develop Nix packages

Instead of building (or realizing) the package (e.g., software, book, data set) with nix-build (or siblings), one will get dropped into a Nix shell with all the dependencies and requirements (e.g., environment) of the package right before it would be built / realized.

That is, if the packages build instructions only ask for make to be executed, then if you issue make on this shell, then the package will be built, and a link will be provided to the result.

0.2.2 Create ad hoc environments

Examples:

developing an application, so you need a couple of scripting language runtimes with packages, debugging tools, a database instance running in the background, etc.
ops work on local/remote VMs (or one just has 12 old ThinkPads lying around) with different operating systems, so firing up a Nix shell with the tools one is used to (of specific versions!)
?

0.2.3 Examples

0.2.3.1 `nix-shell -p`

$ nix-shell -p peek treesheets yt-dlp

will drop you in a shell where the applications peek, treesheets, and yt-dlp are readily available.

0.2.3.2 `nix-shell` with a `default.nix`

TODO: There should be a better example (perhaps a small example repo), but couldn’t come up with anything better this quickly… (Or, at least, add (or redirect to places with) info about -E and callPackage.

Let’s take the treesheets package for example:

$ git clone https://github.com/NixOS/nixpkgs.git
$ cd nixpkgs/pkgs/applications/office/treesheets/
$ nix-shell -E 'with import <nixpkgs> { }; callPackage ./default.nix { }'
$ <the build commands themselves, e.g., cmake>

If you try to execute treesheets in this shell, you’ll get

treesheets: command not found

0.3 Benefits of using Nix shells

Make NixOS / Home Manager configuration lighter by only putting installables there that require extra config.

QUESTION: Although, these could be Nix shellified too, right?
Avoid depending on channels (and thus improving reproducibility) by evaluating the Nixpkgs package collection fetched at a specific commit (or the same with an alternate or one’s own package collection).

QUESTION: Flakes do something similar, don’t they?..
Create a sub-shell with tools for a specific project (e.g., PostgreSQL with config, web framework with all its dependencies)
To work on machines / VMs that are temporary, rarely used, reset periodically, etc., so just install Nix and use nix-shell with the most common tools with the required versions.
?’

0.4 How to start applications from the terminal that won’t close with the terminal?

$ nohup google-chrome-stable & > ~/.nohup.out # or /dev/null
$ disown

See this great answer on nohup vs disown.

1. One-liners with pinning

NOTE: Using the google-chrome package to also demonstrate dealing with unfree packages. See 2.3. Installing unfree packages in the Nixpkgs manual.

ASIDE: tidbits on builtins.fetch* functions
builtins.fetchTarball can handle ZIP archives too:
fetchTarball "https://github.com/NixOS/nixpkgs/archive/<nixpkgs_commit>.zip"
builtins.fetchGit could have been used in the examples below just as well:
fetchTarball "https://github.com/NixOS/nixpkgs/tarball/<nixpkgs_commit>"
translates to
fetchGit {
  url = "https://github.com/NixOS/nixpkgs";
  rev = "<nixpkgs_commit>";
}
^{Some extra reading regarding builtins.fetchGit:
+ (discourse) Difference between fetchgit and fetchGit
+ (gist) an extended, off-the-rails version of the fetchGit documentation
+ (stackoverflow) Git revisions vs references
+ (discourse) rev and ref attributes in builtins.fetchGit (and maybe flakes too?)}

1.1 Get a single package with `nix-shell -p` + `callPackage`

TODO: Add aside about callPackage.

TODO: The Nix reference manual on nix-shell’s -p / --packages doesn’t mention this behaviour and I learned this from @abathur.
update: It is documented in the nix-shell “Examples” section.

NIXPKGS_ALLOW_UNFREE=1                                                                             \
nix-shell -v -p                                                                                    \
'(callPackage                                                                                      \
   (fetchTarball https://github.com/NixOS/nixpkgs/tarball/dc849ffbcd93c2a23e99dcc94efb0962594b8b5f \
   ) {}                                                                                            \
 ).google-chrome'

1.2 Improvise a `shell.nix` on the terminal

NOTE: I believe -E or --expr is only documented in 7.1. Common Options in the Nix manual (or, at least) the only place I saw it in the man pages was in man nix-instantiate, but even there it is just mentioned.)

An ad hoc development sub-shell:

NIXPKGS_ALLOW_UNFREE=1                                             \
nix-shell -v -E                                                    \
"let                                                               \
   pkgs = import (fetchTarball ${NIXPKGS_TARBALL}) {};             \
 in                                                                \
   pkgs.mkShell {                                                  \
     buildInputs = with pkgs; [ google-chrome elixir postgresql ]; \
   }"

1.2.1 `mkShell` vs `mkDerivation`

Older (< 2018) scripts tend to use mkDerivation instead of mkShell, but the latter is only a convenience wrapper around the former when writing Nix expressions for nix-shell. See mkShell needs more documentation in the manual #58624 Nixpkgs issue for more info and links.

1.2.2 Advantages of `mkShell`

(For creating dev environments, that is.)

accepts an inputsFrom list, to “to pull all the dependencies together” from a large project, for example. From the initial PR for mkShell:

[…] you have multiple projects each with their own shell.nix , and you could create 1 master shell.nix that merges all subproject’s shell.nix in case you wanted to work on all the subprojects together. (CMCDragonkai’s comment)

Imagine you have a monorepo with multiple services, each in their own folder with their own default.nix that you can nix-shell and nix-build. On the top-level you can then add a shell.nix with, mkShell { inputsFrom = [ serviceA serviceB ]; to pull all the dependencies together. (zimbatm’s comment)
shellHook s are also composed form input expressions. See PR #63701 for examples.

^{Tons of good info in this Elixir Discourse thread.}

1.3 Nix shell using multiple packages with specific versions (i.e., multiple Nixpkgs revisions)

This comes from this SO answer:

nix-shell -E '
let
  pkgsA = (import (builtins.fetchTarball https://github.com/NixOS/nixpkgs/archive/dfd7183bb1077c064c15575f1dc37a754e13bfc9.tar.gz) {});
  pkgsB = (import (builtins.fetchTarball https://github.com/NixOS/nixpkgs/archive/7d7622909a38a46415dd146ec046fdc0f3309f44.tar.gz) {});
in
pkgsA.mkShell {
  buildInputs = [
    pkgsA.ffmpeg_7-full
    pkgsB.jq
  ];
}'

2. Using `nix-shell` with `shell.nix`

2.1 What is `shell.nix`?

TODO: Come up with a beginner friendly intro that can be embedded later into the official docs, because they are lacking at the moment:

The Nix manual only mentions shell.nix twice, no examples.

The Nixpkgs manual’s Chapter 14. Special builders is probably the best (and only official) one, but it is very minimal. (Also, this was the first time I saw the inputsFrom attribute - every single example I’ve ever encountered used buildInputs thus far.)

There are plenty of examples in the Nixos.wiki, but they are scattered all over the place, and no dedicated entry.

2.2 Issues & solutions

2.2.1 `nix-shell` messes with locales

Had this issue with a more elaborate shell.nix that sets up a Phoenix web project with a PostgreSQL instance (e.g., configures common options, creates tables, does the migrations), and the database kept failing with locale issues.

The solution is from this gist referring to Nixpkgs issue #318: nix-shell sets unsets LANG and friends, specifically to this comment:

The reason it doesn’t work is that you need to set $LOCALE_ARCHIVE, for instance by adding
LOCALE_ARCHIVE = "${pkgs.glibcLocales}/lib/locale/locale-archive";
to the shell expression. (Maybe we should pass LOCALE_ARCHIVE even in --pure mode, but I’m not sure.)

2.2.2 How to make `nix-shell` clean up after itself?

Use trap. It’s a shell built-in; see more on it by typing help trap.

example 1: (unix & linux stackexchange) Killing background processes started in nix-shell
example 2: this

2.x Examples

hellwolf · October 9, 2022, 5:04am

Hi,

Thanks for this.

I am still new to this. But since I am new, I started with nix flake dev shells already. What would be the reason for us new comers using nix-shell for new projects instead of nix flake dev shells?

fricklerhandwerk · October 9, 2022, 8:28am

Pinning nix-shell -p lines is much less cumbersome with -I: Ad hoc shell environments — nix.dev documentation

toraritte · October 9, 2022, 11:23am

Um, there is probably no reason:) I simply started writing about nix-shell because that’s what I ever worked with. Haven’t even tried flakes yet so please consider adding your experiences once this turns into a wiki post!

Thanks! Not sure how I missed this - and there is probably a lot more… Will add it as soon as I can.

Nix shells - tips, tricks, and best practices

0. About Nix shells

0.1 What are Nix shells?

0.2 Use cases (and sources of confusion)

0.2.1 Develop Nix packages

0.2.2 Create ad hoc environments

0.2.3 Examples

0.2.3.1 nix-shell -p

0.2.3.2 nix-shell with a default.nix

0.3 Benefits of using Nix shells

0.4 How to start applications from the terminal that won’t close with the terminal?

1. One-liners with pinning

1.1 Get a single package with nix-shell -p + callPackage

1.2 Improvise a shell.nix on the terminal

1.2.1 mkShell vs mkDerivation

1.2.2 Advantages of mkShell