How do I best use Nix to create a development environment on an HPC cluster without the possibility of system-wide installation?

Help
1

Dear Nix/NixOS community!

I want to use Nix as a package manager for my user-local development environment. Ultimately, what I want to achieve is that I specify a list of packages that I want to have available, run some commands to ensure they are installed and their versions are frozen, and then have the ability to run some other commands to ensure that the paths to directories containing the relevant binaries are in my PATH variable. I would also like to be able to perform an upgrade of all the packages installed in this manner.

My main limitation is that I want to do this all on an HPC cluster, where my permissions are severely limited. As a result, I can’t perform a system-wide installation of Nix, as recommended here. Instead, I chose to use static build of Nix.

Below I describe what I tried, what works, what doesn’t work, and some questions that I have. Ultimately, I would kindly like to ask for your guidance as to which of the potential approaches I should pursue, and how I can make my experience more smooth and productive.

I started by retrieving a static build of Nix from this build, putting it into ~/.local/bin and doing chmod +x ~/.local/bin/nix. Then, I created a ~/.config/nix/nix.conf configuration file with the following contents:

extra-experimental-features = nix-command flakes

The first approach that I tried was to make use of Nix profiles functionality. Using yazi as an example package of interest, I ran nix profile add nixpkgs#yazi, which generated the following persistent output after a very long time cloning and unpacking git objects (please note that the cluster has file amount quotas and works with large file amounts inefficiently):

warning: '/nix/var/nix' does not exist, so Nix will use '/home/sprotser/.local/share/nix/root' as a chroot store
error: cannot read directory "/nix/store/v0sylxg2zng8qbwyfpyphnx8gsc5lzn8-yazi-25.5.31": No such file or directory

Indeed, /nix does not exist, but what is surprising is that attempt was made to access it despite an earlier claim that a user-local store will be used instead. After this command, my PATH variable is unchanged and yazi binary is not available. In my home directory there is .nix-profile, a broken symlink to /home/sprotser/.local/state/nix/profiles/profile. So this approach doesn’t seem to work. Did I do something wrong?

The second approach that I tried is to first run nix run nixpkgs#bashInteractive. After that, my shell prompt to longer shows my user name, showing I have no name! instead. However, both echo $USER and id show all the information as expected. In this shell, /nix exists and is owned by my username, and PATH variable is unchanged. Subsequent execution of nix profile add nixpkgs#yazi nixpkgs#btop completes almost immediately without output, with exit code of 0, but neither yazi nor btop are immediately runnable. However, .nix-profile is now a valid symlink to /home/sprotser/.local/state/nix/profiles/profile, and adding $HOME/.nix-profile/bin to PATH allows me to run both yazi and btop. Note that if I perform the same sequence of actions on another machine, a local virtual machine with a different locale (en_CA.UTF-8), upon running nix run nixpkgs#bashInteractive I get the following output:

bash: warning: setlocale: LC_CTYPE: cannot change locale (en_CA.UTF-8): No such file or directory
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_CA.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_CA.UTF-8): No such file or directory
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_CA.UTF-8): No such file or directory
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_CA.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_CA.UTF-8): No such file or directory

So, this approach somewhat works, except for the fact that the shell fails to show my username in the prompt, doesn’t work well with non-en_US.UTF-8 locales, and the overall idea of using an intermediate shell feels a bit hacky. How do I fix the username and locale issues? Is this approach hacky, or is it perfectly fine?

The third approach that I tried is to exit the previously opened shell, create a flake directory, and in it a flake.nix file with the following contents:

{
  description = "A very basic flake";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
  };

  outputs = { self, nixpkgs }: {

    devShells.x86_64-linux.default = nixpkgs.legacyPackages.x86_64-linux.mkShell {
      buildInputs = [
        nixpkgs.legacyPackages.x86_64-linux.yazi
        nixpkgs.legacyPackages.x86_64-linux.btop
      ];
    };

  };
}

and run nix develop. This command once again takes a long time to complete, processing git objects, and results in persistent output of

warning: creating lock file "/home/sprotser/flake/flake.lock":
• Added input 'nixpkgs':
    'github:nixos/nixpkgs/5e2a59a5b1a82f89f2c7e598302a9cacebb72a67?narHash=sha256-K5Osef2qexezUfs0alLvZ7nQFTGS9DL2oTVsIXsqLgs%3D' (2025-10-19)
error (ignored): package 'nixpkgs#bashInteractive' does not provide a 'bin/bash'

After that, I am in a bash shell, with my username correctly displayed in the prompt, /nix exists and displays as owned by my user id (but not my username as I got with Approach 2. Also, id now only shows my user id, but not my username, as before), and my PATH variable contains a lot of entries, each starting with /nix/store/, followed by what appears to be a build hash and package name, and ending with /bin. yazi and btop are available and run just fine. So this approach also works, however the error (ignored): package 'nixpkgs#bashInteractive' does not provide a 'bin/bash' confuses me. Why doesn’t it provide the requested binary? Should it? Why is this error ignored? Did I do something wrong? I am also confused by the fact that my username doesn’t show up in some places (ls -la /, which I used to check out the /nix directory, and id). How can I fix this? Is it okay that PATH variable now has a lot of entries, one for each package, including dependencies? I plan to install quite a lot of packages, will scaling be a problem?

Finally, after all these operations, ~/.local/share/nix/root/nix/store contains about 200k files, with 6h0sni7pw7gglk3cdml8zmbms1d5k3fq-source and 7agp54mgffm9m1wc1kgmkm37pvy18qhf-source subdirectories each containing about 80k files. Judging by their contents, they seem to be related to the main Nix metadata repository or something like that. Can they be safely removed or packed into archives, so that I don’t have the file amount problem? Under what conditions would I have to unpack them back from the archives? Also, how would I go about ensuring that software versions are frozen until I explicitly request an upgrade, for both approaches 2 and 3? How would I request such an upgrade, for both approaches?

I would greatly appreciate answers to the questions above, as well as any other guidance and insights. Thank you so much!

1 Like
2

@sergeyprotserovca I work for SHARCNET (one of the Canadian HPC providers/alliance members). One of my colleagues pointed me to your post.

You really don’t want the nix unpacked under your home directory. Not only will this kill your file-count quota, it gives very bad performance. The reason is HOME is a shared filesystem and meta-data operations (opening files, closing files, list files, etc.) all have to be globally synchronization across the entire cluster to ensure a consistent filesystem. This makes all directory operations a real bottleneck (creating files, opening files, deleting files, listing files, etc.). Only reading and writing to files once they are open is fast, so you want to work with a few large files opened once, not many small files with lots of opening and closing.

One option is to stick the nix store in a filesystem image (a single large file) and mount that read only (after you have set it up) with apptainer. A potentially better option is to just keep a tarball of your store in your directory and unpack it to local storage on each machine you want to run on. I’ll post a follow up message with an example of how this can be done.

With regard to accessing your local store, nix needs to mount it as /nix. This requires nix to create a new mount namespace, and nix can then only pass this onto decedent processes. It cannot force it onto its parent process. So a new sub-shell is the only option (see here for the docs about how a local store works).

I would also personally recommend using the nix shell command to spawn a new sub-shell with /nix mounted and PATH extended and not nix develop unless you actually want a development environment.

nix shell nixpkgs#yazi nixpkgs#btop

Not sure I follow exactly what your doing/what is going on with your locales, but possibly you should be using C.UTF-8 as this is the most basic locale. Currently there is a regression on the latest nibi (the supercomputer SHARCNET manages) image so LANG is no longer set. This results is come programs, like btop no longer running. You can work around this by manually setting it yourself export LANG=C.UTF-8.

I would also note, that the SHARCNET systems (formerly graham and now nibi) do have a global nix install. The new nibi one doesn’t yet have a user accessibly nix-daemon, but that is on my todo list. It doesn’t seem likely though that the non-SHARCNET systems (fir, trillium, rorqual, etc.) will have a global nix install anytime soon. So you still need to use the local install method on them.

1 Like
3

This post originally suggested using nix shell by itself. That isn’t great as nix shell always installs the latest. So it was updated to use a flake as discussed in later comments.

Some global config

We want the nix command, flakes, and the registry tarball time-to-live set to something much longer than the default hour as downloading and unpacking into the cache in HOME is very slow

mkdir --parent ~/.config/nix
cat > ~/.config/nix/nix.conf <<EOF
extra-experimental-features = nix-command flakes
tarball-ttl = $((7*24*60*60))
EOF

Create a local nix store

We need a static copy of nix so it can run without /nix mounted (a non-static copy’s libraries won’t resolve without /nix mounted).

mkdir -p ~/.local/bin
curl -L https://hydra.nixos.org/job/nix/maintenance-2.32/buildStatic.nix-cli.x86_64-linux/latest/download-by-type/file/binary-dist -o ~/.local/bin/nix
chmod +x ~/.local/bin/nix

The store needs to be on local storage for speed. We specify this with the environment variable NIX_CONFIG instead of ~/.config/nix.conf as the location will change each time we unpack it

NIX_STORE=$(mktemp -dp /local/tmp)
export NIX_CONFIG="store = local:$NIX_STORE"

NIX_STORE is just for our use later when taring it up and cleaning up.

Picking a nixpkgs release

With just nix on a non-NixOS system, nixpkgs defaults to nixpkgs-unstable

nix registry list

I usually like to set it to a NixOS release, as what happens on a NixOS system, for a bit more stability

nix registry add nixpkgs github:NixOS/nixpkgs/nixos-25.05

Declare an environment in a flake

As discussed later in this thread, it is likely best to declare your desired packages in a flake to fix the dependency versions and be able to put everything in git

mkdir myproject
cd myproject
git init

The simplest flake has implicit inputs and uses buildEnv to create a profile-like directory of symlinks to the files of the listed packages

cat >flake.nix <<EOF
{
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = with nixpkgs.legacyPackages.x86_64-linux; buildEnv {
      name = "myenv";
      paths = [
        yazi
        btop
      ];
    };
  };
}
EOF
git add flake.nix

Doing a nix build on it pins the flake inputs, put it in the store (if available, non-X11 dependent versions of packages will help reduce the store size), creates an auto garbage-collection (gc) root for it (so it isn’t deleted when gcing), and creates a symlink for using it

nix build .# -o myenv

Saving the local nix store

GCing minimizes the store by removing everything without a gc root (such as any build-only dependencies)

nix store gc

Then we can pack it up into a single file in HOME

tar -czf nix-store.tar.gz -C $NIX_STORE .

This is perfect as efficient usage of HOME required just reading and writing a few large files. Deleting the unpacked version requires adding write permission in the intermediate directories

chmod -R u+rwX $NIX_STORE
rm -fr $NIX_STORE

Restoring and using the local nix store

Anytime we want to use it, we just create a directory on the local storage

NIX_STORE=$(mktemp -dp /local/tmp)
export NIX_CONFIG="store = local:$NIX_STORE"

and unpack it there again

tar -xzf nix-store.tar.gz -C $NIX_STORE .

Then nix shell on the nix build symlink will give a sub-shell with /nix mounted and and PATH extended for the build output

nix shell ./myenv

Running nix shell .# works too, but it also sucks a bunch of evaluation stuff back into the store.

Using the local nix store in a job

Using it inside a job

salloc

is the same. SLURM_TMPDIR is set to local storage (if it exists), so unpack the store there

NIX_STORE=$(mktemp -dp $SLURM_TMPDIR)
export NIX_CONFIG="store = local:$NIX_STORE"
tar -xzf nix-store.tar.gz -C $NIX_STORE

Then, for interactive usage, use nix shell to get a sub-shell again

nix shell ./myenv

or specify a command to run something non-interactively

nix shell ./myenv --command MYPROGINMYENV

There is no need to clean it up, as the scheduler will take care of that when the job ends.

4

@twhitehead, thank you so much for your response!

I am actually on Killarney, which does not have a system-level Nix installation. In addition, I am working on another HPC cluster which is managed by a different organization, and doesn’t have a system-level Nix installation either, so my questions are not very specific to a particular cluster. Ideally, I would like to have a single configuration that is as cluster-agnostic as possible.

Talking about the “many small files” problem, I am thinking of using Nix to install just a few programs. In fact, right now on Killarney I just downloaded each of them and unpacked them to $HOME manually. This works, but is not very ergonomic. As I understand, some of the operations with Nix involve cloning a remote repository, which results in a lot of files being created. I fully intend to avoid running such operations often. However, I see that even without these repository copies, installing just two programs resulted in about 40k files being created. It may be the case that Nix (outside of Apptainer) is just not a good fit for my case, and I should either consider using Apptainer (it is available on Killarney, but not on the other cluster I use), or just roll with manual program installation: they are not too many, after all.

My usecase for these additionally-installed programs is for them to complement my main working environment on the login nodes. They include editors, file managers and such, so they are not required on job nodes. I am going to check if there is user-specific local storage on the login nodes of interest, and what the spatial constraints for it are, and maybe I will be able to use the packing approach you suggested.

Could you please elaborate on why the nix shell approach is better than nix develop? I thought the latter is closer to letting me install some stuff at once, freeze the versions and update them only if needed. I am not sure how this works with nix shell, but I may be missing something.

Thank you again!

5

Nix develop kicks you into an environment intended to build the package you run it on. This has a lot of baggage in it if you are just wanting to run some programs.

For example, on my home machine I have around 2412 lines of bash function declared and 125 lines of environment variables, and my PATH has 8 components

[tyson@tux:~/example]$ declare -fp | wc -l
2412
[tyson@tux:~/example]$ declare -p | wc -l
126
[tyson@tux:~/example]$ echo $PATH | tr : '\n' | wc -l
8

When I use your earlier flake to get yazi and btop on the PATH by entering a development environment where they are buildInputs, this goes to 4209 line of bash declared functions, 232 lines of environment variables, and a PATH with 31 components

[tyson@tux:~/example]$ nix develop .#
[tyson@tux:~/example]$ declare -fp | wc -l
4209
[tyson@tux:~/example]$ declare -p | wc -l
232
[tyson@tux:~/example]$ echo $PATH | tr : '\n' | wc -l
31

If I use nix shell I only get one more component in my PATH for each package I ask for

[tyson@tux:~/example]$ nix shell nixpkgs#yazi nixpkgs#btop
[tyson@tux:~/example]$ declare -fp | wc -l
2412
[tyson@tux:~/example]$ declare -p | wc -l
126
[tyson@tux:~/example]$ echo $PATH | tr : '\n' | wc -l
10
6

If you want to lock down version of programs with a flake (which is a good idea), go with a buildEnv

cat >flake.nix <<EOF
{
  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = with nixpkgs.legacyPackages.x86_64-linux; buildEnv {
      name = "project";
      paths = [
        yazi
        btop
      ];
    };
  };
}
EOF

and use nix shell to access it

[tyson@tux:~/example]$ nix shell .#
[tyson@tux:~/example]$ declare -fp | wc -l
2412
[tyson@tux:~/example]$ declare -p | wc -l
126
[tyson@tux:~/example]$ echo $PATH | tr : '\n' | wc -l
9
7

nix profile is decent too. It isn’t something like a flake that you can check into git though. And while you can use --profile to specify a path in order to have multiple profiles, it is really best used to add a few packages you always want available in a your global profile.

The other thing is I couldn’t figure out anyway to get nix to spawn me a sub-shell with /nix mounted so I could use the profile without specifying a package to nix shell ... or nix run .... Would be nice if your could do nix shell with no package specified to just got a sub-shell with/nix mounted. Your nix run nixpkgs#bashInteractive seems like a decent workaround though.

Not sure why you were getting all the locale issues with nix run nixpkgs#bashInteractive. It seemed to work fine in my testing. Possibly try the C.UTF-8 locale?

So, yeah, you are right. Probably best to use a flake to lock down your versions or a profile. The get-the-latest behaviour of nix shell isn’t really what you want.

8

Is it not possible to use nix-portable?

1 Like
9

@WeetHet thanks for the link. I haven’t tried nix-portable myself, but the under the hood section on the main page strongly implies it is going to be placing a lot of small files in HOME, which is something @sergeyprotserovca was looking to avoid (due to the file count quotas and the bad performance)

  • The nix-portable executable is a self extracting archive, caching its contents in $HOME/.nix-portable
  • Either nix, bubblewrap or proot is used to virtualize the /nix/store directory which actually resides in $HOME/.nix-portable/store
10

Thank you so much everyone for your responses! Given all the new information, I am going to explore how well I can make this work in my environments, and will post here if I run into something problematic or interesting :slight_smile:

11

So I have switched from the master branch of Nix builds (the build link is included in the very first post I made in this topic) to the maintenance-2.32 branch and the latest build, and your suggested approach with buildEnv and nix shell works just flawlessly, thank you so much! The problem of many small files being stored in my $HOME is still there, but it is somewhat orthogonal to the other problems discussed here, and I will figure it out on a cluster-specific basis. I therefore mark your answer as the solution, and thank you again!

12

That is good to hear. I went back and revised my earlier answer to reflect all the improvements we made (mostly using flakes in order to tie the down the revision instead of just nix shell, but also some stuff nix build and garbage collect to compact the store). Hopefully that makes it a bit easier for anyone else should they come along with everything in one post.

Using the latest nix from the latest maintenance release instead of the bleeding edge master is a solid suggestion too. I’ll update my instruction to use maintenance-2.32 too. Thanks!

You can also try and see if whatever package you wants build as a static package. For example

nix build nixpkgs#pkgsStatic.btop

Often it doesn’t succeed (as with btop), but if it does, then you just have a static binary with no dependencies.

1 Like