`writeTextFile`: How does the `meta` input attribute get its default values and where does `textPath` come from?

toraritte · October 18, 2022, 10:25pm

Just looked at the current implementation of writeTextFile,

source of `writeTextFile`:

  writeTextFile =
    { name # the name of the derivation
    , text
    , executable ? false # run chmod +x ?
    , destination ? ""   # relative path appended to $out eg "/bin/foo"
    , checkPhase ? ""    # syntax checks, e.g. for scripts
    , meta ? { }
    }:
    runCommand name
      { inherit text executable checkPhase meta;
        passAsFile = [ "text" ];
        # Pointless to do this on a remote machine.
        preferLocalBuild = true;
        allowSubstitutes = false;
      }
      ''
        target=$out${lib.escapeShellArg destination}
        mkdir -p "$(dirname "$target")"

        if [ -e "$textPath" ]; then
          mv "$textPath" "$target"
        else
          echo -n "$text" > "$target"
        fi

        eval "$checkPhase"

        (test -n "$executable" && chmod +x "$target") || true
      '';

and couldn’t figure out these:

QUESTION 1. Where does `textPath` come from?

Probably missed a with statement or something otherwise obvious, but it just drives me crazy.

ANSWER

As @abathur points out below, I totally missed the passAsFile attribute that can be passed to a derivation:

A list of names of attributes that should be passed via files rather than environment variables.

Follow-up to QUESTION 1: Why is there a need then to check if `textPath` exists?

The body of the writeTextFile function has the following check:

        if [ -e "$textPath" ]; then
          mv "$textPath" "$target"
        else
          echo -n "$text" > "$target"
        fi

Doesn’t passAsFile ensure that this file will always be created? At least, I couldn’t see any write* functions building on writeTextFile tamper with this one (and don’t even know how they could).

ANSWER

From @YorikSar:

As for the check whether textPath exists, I would guess that it was written when the support for passAsFile was just added to Nix, and the author wanted to support older versions of Nix as well as the new one. And here’s the PR that introduced that: writeTextFile: Use passAsFile if available by wmertens · Pull Request #6411 · NixOS/nixpkgs · GitHub - indeed, it was added even before passAsFile support was released.

QUESTION 2. How does the `meta` input attribute get its default attributes?

^{Just learned about meta-attributes while writing this question and looking up the suggested threads in the side panel…}

writeTextFile’s input attribute set has a meta attribute that defaults to {}, but it is not used anywhere in its body. writeTextFile calls runCommand, but none of the runCommand* funcitons touch meta explicitly. The only place meta is used (in relation with writeTextFile) is in writeShellApplication:

meta.mainProgram = name;

Attributes cannot be set this way,

nix-repl> as = { a = 1; b = 3; }

nix-repl> as.c = 27
error: syntax error, unexpected '=', expecting end of file

       at «string»:1:6:

            1| as.c = 27
             |      ^

so where is it set?

ANSWER

(The wrong path I went down on - leaving this here for educational purposes)

@YorikSar pointed out below in an answer that this is wrong, but leaving this here just in case.

Did a brute-force grep on 'mainProgram = [^"]' (hoping that the default value won’t be an empty string…), and the most generic result not tied to any specific packages was in pkgs/stdenv/generic/check-meta.nix. Following the breadcrumbs in this file,

mainProgram = str;
← metaTypes
← checkMetaAttr
← checkMeta
← (kind of lost the thread here, but I think checkMeta is then called in the function body)

it feels like this is the place that sets the missing attributes. Started looking if pkgs/stdenv/generic/check-meta.nix is called anywhere meaningful, and the answer is yes:

pkgs/stdenv/generic/make-derivation.nix:6
← pkgs/stdenv/generic/default.nix:55
← mkDerivation
← writeShellApplication
← writeTextFile
← runCommand
← runCommandWith
← (calls stdenv.mkDerivation)

The weakest part of this is that check-meta is quite complex and I simply intuited that the defaults are set there…

@YorikSar’s first answer below explains where meta default attributes are set, but it turns out that my question is unrelated to that; @YorikSar’s second answer pointed out that I did not consider the scope, and thus made false assumptions:

meta.mainProgram = name;

is used like this in writeShellApplication:

  writeShellApplication =
    { # ...omitted...
    }:
    writeTextFile {
      # ...omitted...
      meta.mainProgram = name;
    }

where the attribute set is defined to be an input for writeTextFile. As the input attribute set is just being constructed, one can apply the Nix convention to define nested attribute sets more succinctly, so the above example can be re-written as (pasting @YorikSar 's example verbatim):

  writeShellApplication =
    { # ...omitted...
    }:
    writeTextFile {
      # ...omitted...
      meta = {
        mainProgram = name;
      };
    }

In nix repl:

nix-repl> ({ a.b = 27; }).a.b
27

abathur · October 18, 2022, 11:52pm

https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-passAsFile

YorikSar · October 19, 2022, 6:28am

As you correctly noticed, meta attribute is passed to writeTextFile and defaults to empty attribute set. Then it’s passed via inherit to runCommand that is a wrapper around runCommandWith in derivationArgs attribute set.

  runCommandWith =
    let
      # prevent infinite recursion for the default stdenv value
      defaultStdenv = stdenv;
    in
    { stdenv ? defaultStdenv
    # which stdenv to use, defaults to a stdenv with a C compiler, pkgs.stdenv
    , runLocal ? false
    # whether to build this derivation locally instead of substituting
    , derivationArgs ? {}
    # extra arguments to pass to stdenv.mkDerivation
    , name
    # name of the resulting derivation
    # TODO(@Artturin): enable strictDeps always
    }: buildCommand:
    stdenv.mkDerivation ({
      enableParallelBuilding = true;
      inherit buildCommand name;
      passAsFile = [ "buildCommand" ]
        ++ (derivationArgs.passAsFile or []);
    }
    // (lib.optionalAttrs runLocal {
          preferLocalBuild = true;
          allowSubstitutes = false;
       })
    // builtins.removeAttrs derivationArgs [ "passAsFile" ]);

Here runCommandWith calls stdenv.mkDerivation and composes arguments for it. meta isn’t mentioned anywhere, so in last line it remains in the attribute set after removeAttrs and is added to the resulting attribute set via //. Now meta attribute (still empty by default) is passed to stdenv.mkDerivation.

mkDerivation comes from make-derivation.nix, and it indeed includes checkMeta, but that is used only to check if meta is valid, not to generate defaults. It is created in the same file:

  # The meta attribute is passed in the resulting attribute set,
  # but it's not part of the actual derivation, i.e., it's not
  # passed to the builder and is not a dependency.  But since we
  # include it in the result, it *is* available to nix-env for queries.
  meta = {
      # `name` above includes cross-compilation cruft,
      # is under assert, and is sanitized.
      # Let's have a clean always accessible version here.
      name = attrs.name or "${attrs.pname}-${attrs.version}";

      # If the packager hasn't specified `outputsToInstall`, choose a default,
      # which is the name of `p.bin or p.out or p` along with `p.man` when
      # present.
      #
      # If the packager has specified it, it will be overridden below in
      # `// meta`.
      #
      #   Note: This default probably shouldn't be globally configurable.
      #   Services and users should specify outputs explicitly,
      #   unless they are comfortable with this default.
      outputsToInstall =
        let
          hasOutput = out: builtins.elem out outputs;
        in [( lib.findFirst hasOutput null (["bin" "out"] ++ outputs) )]
          ++ lib.optional (hasOutput "man") "man";
    }
    // attrs.meta or {}
    # Fill `meta.position` to identify the source location of the package.
    // lib.optionalAttrs (pos != null) {
      position = pos.file + ":" + toString pos.line;
    } // {
      # Expose the result of the checks for everyone to see.
      inherit (validity) unfree broken unsupported insecure;
      available = validity.valid != "no"
               && (if config.checkMetaRecursively or false
                   then lib.all (d: d.meta.available or true) references
                   else true);
    };

Here some attributes can be overridden by meta from arguments (name, outputsToInstall), some are generated by checkMeta via validity variable (unfree, broken, …).

That’s how even the very simplest derivation gets meta attribute like this:

nix-repl> pkgs = import <nixpkgs> {} 

nix-repl> :p (pkgs.stdenv.mkDerivation {name = "hello";}).meta 
{ available = true; broken = false; insecure = false; name = "hello"; outputsToInstall = [ "out" ]; position = "(pkgs.stdenv.mkDerivation {name = \"hello\";}).meta:1"; unfree = false; unsupported = false; }

toraritte · October 19, 2022, 10:57am

@abathur Thanks! Totally missed this one… I still don’t get one thing though: why is there a need to check if textPath exists? Doesn’t passAsFile ensure that this file will always be created? At least, I couldn’t see any write* functions building on writeTextFile tamper with this one (and don’t even know how they could).

        if [ -e "$textPath" ]; then
          mv "$textPath" "$target"
        else
          echo -n "$text" > "$target"
        fi

@YorikSar Wow, it would have taken me a long while to find out that meta defaults are actually set in make-derivation.nix… The nix repl demonstration was especially great. Thank you!

(I still don’t get how mainProgram is set or why the evaluation doesn’t throw when it gets there by a quick glance on that code block, but it is going to be fun to decipher it, now that I know where to look - and that I don’t have to grep the entirety of Nixpkgs for clues:)

YorikSar · October 19, 2022, 12:31pm

writeShellApplication doesn’t participate in the writeTextFile chain, and it doesn’t accept meta argument.
This syntax, however, is quite popular in Nix. { meta.mainProgram = name; } is equivalent to:

{
  meta = {
    mainProgram = name;
  };
}

It doesn’t update any existing variables (they are immutable in Nix), but rather defnes nested attributes in the set.

YorikSar · October 19, 2022, 12:34pm

As for the check whether textPath exists, I would guess that it was written when the support for passAsFile was just added to Nix, and the author wanted to support older versions of Nix as well as the new one.

toraritte · October 19, 2022, 1:01pm

Thanks for confirming my textPath theory and for the simple meta.mainProgram explanation! I got into a sort of a tunnel vision regarding the latter, and didn’t consider the context, but kept obsessing on what I thought was going on…

YorikSar · October 19, 2022, 1:28pm

Oh, here’s the PR that introduced that: writeTextFile: Use passAsFile if available by wmertens · Pull Request #6411 · NixOS/nixpkgs · GitHub - indeed, it was added even before passAsFile support was released.

toraritte · October 19, 2022, 1:48pm

Thanks so much! How did you find this? git blame?

YorikSar · October 19, 2022, 2:09pm

Yes. I opened blame in GitHub UI (three dot menu for file, “view blame”).

`writeTextFile`: How does the `meta` input attribute get its default values and where does `textPath` come from?

QUESTION 1. Where does textPath come from?

ANSWER

Follow-up to QUESTION 1: Why is there a need then to check if textPath exists?

ANSWER

QUESTION 2. How does the meta input attribute get its default attributes?

ANSWER

QUESTION 1. Where does `textPath` come from?

Follow-up to QUESTION 1: Why is there a need then to check if `textPath` exists?

QUESTION 2. How does the `meta` input attribute get its default attributes?