Refrencing files with ./path/to/foo - am I holding it wrong?

Help
1

This one got me off guard. Been on nixos for several years now, and I always
relied on files referenced by ./path/to/foo.env being available on the
target machine.

tl;dr - I believe there’s a few potential non-obvious things going on in here:

  • files referenced with ./path/to/foo.env not being copied to remote
  • nixos VMs opaquely accessing host’s /nix/store
  • oci-containers’s environmentFiles too lazily evaluated (or not at all)

The other day I was setting up managed containers via oci-containers.
They rely on certain environment files in order to run.

The part that made this so hard to debug is - locally
everything works ok, but when pushed to a destination system (e.g. via
nixos-rebuild --target-host), .foo.env is not available.

It turns out that .foo.env is availble in the builder’s nix store, and when a
local VM is run via nixos-rebuild -f .#foo build-vm && ./result/bin/run-foo-vm,
the VM sees the file and works OK. Logging into the VM and manually checking
/nix/store path seems like the file was copied onto the VM, while in fact it’s
referencing the builder’s /nix/store.

For example, this is a barebone VM with fd as the only installed package,
and it’s finding cliamp in it’s nix store.

[alice@foo:~]$ fd cliamp /nix/store
/nix/store/y019w1npngzswvz94hl99bvghm55zb9v-system-path/bin/cliamp
/nix/store/wfq3f1cli2n02vcx3w913dkn1g174d3k-cliamp-1.39.1.drv
/nix/store/b3aa8mw38wr2c2w3wi1y22j92kvk287g-cliamp-1.35.0-go-modules.drv
...

Repro

Create an env file we’ll use:

$ echo FOO=bar > .foo.env

If in a git repo, git add .foo.env.
Add this module to a nixos machine:

virtualisation.oci-containers.containers = {
  hello = {
    cmd = [
      "sh"
      "-c"
      "env | grep FOO || echo FOO is missing!"
    ];
    environmentFiles = [
      # ../.foo.env # <- file doesn't exist, flake builds no problem.
      # ./doesnt-exist.env # self explanatory. builds normally
      ./.foo.env # <- file exists. TODO check if service can read it
    ];
    image = "docker.io/library/alpine:latest";
  };
};

Quick REPL check (foo is the VM, remotemachine is a real remote machine):

nix-repl> outputs.nixosConfigurations.foo.config.virtualisation.vmVariant.virtualisation.oci-containers.containers.hello.environmentFiles
[ /nix/store/1y2krcz9z3ds5w68j4ag0d6g4cniax5m-source/.foo.env ]

nix-repl> outputs.nixosConfigurations.remotemachine.config.virtualisation.oci-containers.containers.hello.environmentFiles
[ /nix/store/1y2krcz9z3ds5w68j4ag0d6g4cniax5m-source/.foo.env ]

Build and run the VM:
$ nixos-rebuild --flake .#foo build-vm && ./result/bin/run-foo-vm -nographic

Check for FOO in the VM:

[alice@foo:~]$ journalctl -eu podman-hello | grep -E 'FOO|Error'
Aug 05 21:06:57 foo hello[702]: FOO=bar

Build and deploy the same module to the remote machine:
$ sudo nixos-rebuild --flake .#remotemachine --target-host remotemachine switch

$ ssh remotemachine  "journalctl -eu podman-hello | grep -E 'FOO|Error'"
Aug 05 16:55:19 mx-001 podman-hello-start[8599]: Error: parsing file "/nix/store/1y2krcz9z3ds5w68j4ag0d6g4cniax5m-source/.foo.env": open /nix/store/1y2krcz9z3ds5w68j4ag0d6g4cniax5m-source/.foo.env: no such file or directory

Expected

Referencing files with ./path/to/foo.env copies them over to the target host.

2

Try "${./.foo.env}". I think the fact that it works in the local case is an accident — because you’re using flakes, your whole source is being copied into the store and .foo.env is coming along for the ride. The flake source isn’t considered a dependency of the built system, so it isn’t copied to the remote machine when the system is deployed. Interpolating a path into a string gets it put into the store individually, and that store path then becomes part of that string’s context and is tracked as a dependency of your built derivation.

General rule of thumb: use bare paths if the file contains data you need at eval time, push paths into strings if the file contains data you need at build or run time.

2 Likes
3

While the string antiquote method gets around this, it’s clearly a bug in either nix or nixpkgs. The string context is getting lost on its way to a derivation somewhere. That should never happen.

4

With low-medium confidence, I think it’s a language design bug — something we could change if we were willing to version (or introduce breaking changes to) the behavior of paths in the language, but not otherwise.

5

Well, if someone used toString somewhere they shouldn’t, you could partially blame that on language design. The only other potential causes I can see are using unsafeRemoveStringContext, which is dangerously named enough that I can’t call that a language design bug, just a bad programmer, or an actual nix behavioral bug where string context isn’t tracked where it should be, in which case fixing it is the right move even if it changes behavior.

6

Yes, that’s the mechanism here. The path is going to lib.escapeShellArg, which uses toString to coerce its input.

I guess something we could change that wouldn’t involve changing the language would be for lib.types.path to only accept actual string values, not path values; or, less globally disruptively, for this option to use a more restrictive type.

7

Then the code that passes it to lib.escapeShellArg should do a "${...}" around it before passing it, right?

8

Hmm, probably yeah, though I’d want to think about it for longer than I have to be sure that there isn’t some valid case for using an eval-time path there. I’m not nearly familiar enough with this module and its usage patterns, beyond what a quick skim of its contents has told me.

9

Thanks @rhendric, that indeed does work. I’m still baffled though, don’t think this is either well documented or a good user experience. Agreed with @tejing on the points above.

Thinking about this, I’d expect strings to have these issues, not paths.
Paths should be a hard reference.

10

Using toString on a path-type is always wrong. (Unless you’re messing with things at the level of unsafeDiscardStringContext and you’re absolutely sure the end result gets the correct string context.) If it was meant to be resolved at runtime, it should have been a string from the beginning, not a path.

1 Like
11

I certainly agree that it’s always wrong for new code to do that. It doesn’t necessarily follow that nobody’s expectations are broken if you change all old code to stop doing that.

(I’m being pedantic; I do think it’s probably safe in this case to force the interpolation in the module.)

12

If that’s a big enough issue to warrant it, the thing to do would be to rig the module to warn if a path-type is passed, then change it when enough time has passed. It’s not an argument not to change it.

2 Likes