`specialArgs` in tests?

I developed a system to automatically generate hardening configuration for
systemd services^1.

The idea is to clone nixpkgs, and instrument each test using an object I called
systemdPassthru (I am very open to suggestions for better names), that I can use
to manipulate configuration options.

In practice, a service like this

{ pkgs, ...}:
{
  options = ...;
  config.systemd.services.myservice.serviceConfig = {
    Foo = "Bar";
  };
}

becomes this

{ pkgs, systemdPassthru ...}:
{
  options = ...;
  config.systemd.services.myservice.serviceConfig = {
    Foo = "Bar";
    PrivateDevices = systemdPassthru.myservice.PrivateDevices;
  };
}

This transformation is done autmatically by a tool I wrote in Rust using rnix^2.

In order to make this work, I have do modify two things in nixpkgs:

*** nixpkgs/nixos/tests/make-test-python.nix
--- nixpkgs/nixos/tests/make-test-python.nix -- "hooked" version
***************
*** 1,9 ****
  f: {
    system ? builtins.currentSystem,
    pkgs ? import ../.. { inherit system; },
    ...
  } @ args:
  
! with import ../lib/testing-python.nix { inherit system pkgs; };
  
  makeTest (if pkgs.lib.isFunction f then f (args // { inherit pkgs; inherit (pkgs) lib; }) else f)
--- 1,10 ----
  f: {
+   systemdPassthru,
    system ? builtins.currentSystem,
    pkgs ? import ../.. { inherit system; },
    ...
  } @ args:
  
! with import ../lib/testing-python.nix { inherit system pkgs; specialArgs = { inherit systemdPassthru; }; };
  
  makeTest (if pkgs.lib.isFunction f then f (args // { inherit pkgs; inherit (pkgs) lib; }) else f)
*** output/nixpkgs/nixos/tests/all-tests.nix
--- output/nixpkgs/nixos/tests/all-tests.nix -- "hooked" version
***************
*** 1,4 ****
! { system, pkgs, callTest }:
  # The return value of this function will be an attrset with arbitrary depth and
  # the `anything` returned by callTest at its test leafs.
  # The tests not supported by `system` will be replaced with `{}`, so that
--- 1,4 ----
! { systemdPassthru, system, pkgs, callTest }:
  # The return value of this function will be an attrset with arbitrary depth and
  # the `anything` returned by callTest at its test leafs.
  # The tests not supported by `system` will be replaced with `{}`, so that
***************
*** 15,21 ****
      else if hasAttr "test" val then callTest val
      else mapAttrs (n: s: discoverTests s) val;
    handleTest = path: args:
!     discoverTests (import path ({ inherit system pkgs; } // args));
    handleTestOn = systems: path: args:
      if elem system systems then handleTest path args
      else {};
--- 15,21 ----
      else if hasAttr "test" val then callTest val
      else mapAttrs (n: s: discoverTests s) val;
    handleTest = path: args:
!     discoverTests (import path ({ inherit systemdPassthru system pkgs; } // args));
    handleTestOn = systems: path: args:
      if elem system systems then handleTest path args
      else {};

But some tests don’t work well with this pattern (54 out of 564).

To make systemdPassthru work, I used specialArgs.
I’m wondering if the right thing to do is to make every single test accept specialArgs.
This would allow for some modularity in test instantiation:

let myTests = import <nixpkgs/nixos/tests/all-tests.nix> {
    specialArgs = {
      # Options I pass to some module that expects some special config.
    };
    system = builtins.currentSystem;
    pkgs = import <nixpkgs> { system = builtins.currentSystem; };
    callTest = t: lib.hydraJob t.test;
  };

This is very useful in my case, but even I think the usefulness of this pattern is
arguable outside the scope of my work.

I’m also wondering if the right approach isn’t actually to use _module instead of specialArgs.

In any case, I have to make my generation system work, so I’m looking for some advice
to choose between these options:

  • Either do this on my side, make a local branch where I modify the 54 misbehaving tests
    and use git mail to generate pathches, and add a patch phase to
    the workflow of nixos-harden-systemd
    This option would trivially work, but it does not feel very satisfactory, at least to me.
  • Do about the same thing, but make every test accept specialArgs, and commit the result
    in nixpkgs. Then derive my special case from this.
    Or exactly that, but with _module instead of specialArgs, or with both _module and specialArgs.

What does feel right to you?
I’m of course open to any other suggestion, and willing to discuss this further and give
more information / explain any unclear point of what I’ve done.

This project is funded by the European Commission through nlnet, and I am really grateful to them for the opportunity!

2 Likes

I’m also wondering if the right approach isn’t actually to use _module instead of specialArgs .

Yes. See the comment here:

                , # This should only be used for special arguments that need to be evaluated
                  # when resolving module structure (like in imports). For everything else,
                  # there's _module.args. If specialArgs.modulesPath is defined it will be
                  # used as the base path for disabledModules.

It looks like a neat little idea to have defaults for various systemd service options but why make it a module argument? Why not have a systemd.defaultService.serviceConfig.* which can be set to harden everything by default?

There probably will be some extra problems with that approach but they’re all implementation and not inherently flawed design. For example there are still a few packages that take their configruation via the global nixpkgs config (import <nixpkgs> { this = "one"; }) instead of override which makes things

1 Like

Ok, thanks!

To summarize what I want to obtain:
The ultimate goal is to be able to instantiate a given test with an arbitrary configuration. This looks like that:

let
  defaultPassthru =
    # ./output/tests contains the name of each systemd service and for each service
    # the list of the fields that have been hooked
    let tests = builtins.fromJSON (builtins.readFile ./output/tests);
    in lib.mapAttrs (_: value: builtins.listToAttrs (map (name: lib.nameValuePair name false) value.fields)) tests;
  lib = (import <nixpkgs> {}).lib;

  allTestsHooked = import (toString ./output/nixpkgs/nixos/tests/all-tests.nix) {
    systemdPassthru = defaultPassthru // {
      # This is where I need the magic to happen:
      # I want instantiate the tests *with a given set of hardening options*
      "hydra-server" = defaultPassthru."hydra-server" // {
        "ProtectKernelTunables" = true;
      };
    };
    system = builtins.currentSystem;
    pkgs = import (toString ./output/nixpkgs) { system = builtins.currentSystem; };
    callTest = t: lib.hydraJob t.test;
  };
in allTestsHooked.hydra.hydra-unstable

The problem is to pass some global config to every system that will be generated by the tests.
In any case, I need some way to “speak through” the test top-level (nixos/tests/all-tests.nix, nixos/lib/make-test-pyton.nix) down to the nixos modules that I instrumented.

If I understand correctly, you are suggesting to instead pass my arguments through the top-level nixpkgs that I pass to nixos/tests/all-tests.nix?


In more detail, the problems that I encountered, which will hopefully make my train of thought clearer (I don’t want to sound argumentative, I’m just genuinely stuck with my design issues haha):

It looks like essentially, the only point where I can pass any special to the modules that are instantiated is here in nixos/lib/testing-python.nix. The only external variables that are available there come from here (the usable fields are pkgs, config, and specialArgs). If I want to use specialArgs, I need each test to correctly pass any specialArgs I give to it (which does not trivially happen in the 54 cases I mentioned), same goes for config if I remember correctly. Then the only way to pass any argument to the modules without modifying any test is to add some special fields to pkgs which is correctly passed through the whole call hierarchy (hopefully). But then, isn’t using pkgs a bit of a dirty trick? (Well, passing specialArgs is kind of a dirty trick too, I can’t figure out which is better. Using pkgs, which I didn’t think of, has the advantage of being painless.)

The “dual” question is: Will anyone need to pass something to every module like I’m doing right now, or is my use case too special? (Another question: Do we actually want to be able to do that?)

1 Like

[Not quoting anything since I’m replying to almost all of it]

I meant that indeed arbitrary pass through across NixOS systems (even the test ones!) for evaluation is probably bad since the nixosTests already do their job for nixpkgs’s usecase.

Looking a bit closer testing-python.nix already has an extraConfigurations argument and it seems reasonable to me that you could use that to pass things through. You can make a module that sets all the hardening options you want and pass it through to makeTest. If you’re not calling that directly I guess you should make the caller of makeTest expose extraConfigurations and PR that which seems like a significantly simpler change.

I’m improving the test wiring with the module system:

https://github.com/NixOS/nixpkgs/pull/176557

To add your own configuration to an existing test, you can use the new extend function for tests. defaults and nodes work just like in NixOps.

nixosTests.foo.extend {

  # set any test-level options here, such as `enableOCR`, `meta`, ...

  defaults = { config, lib, ... }: {
    # set any NixOS-level options here, applied to all VMs
  };

  nodes.foo = {
    # set any NixOS-level options here, applied to the `foo` machine only
    # (created if necessary)
  };
}

If nixosTests.foo consists of multiple test cases and/or a test matrix, i.e. multiple derivations, it works exactly the same, requiring no mapAttrs or anything like that.

2 Likes

@ckie Yes, passing extraConfiguration seems like a reasonable approach and it seems “PRable” :slight_smile: anyway it solves my issues without too much trouble, thanks!

@roberth your solution is brillant! Is it stable enough for me to base my branch on yours? Or should I use the other solution (which works just fine too for me) while waiting for the final merge?

I expect it to be merged no earlier than two weeks from now.
That may be optimistic considering the obvious dependency on volunteer maintainers. I know the test framework maintainer is positive about the idea, but hasn’t dived into it yet.

I wouldn’t mind if you wired extraConfigurations into all-tests.nix. I can support it in my own PR and when it becomes conceptually redundant, we may deprecate it. The latter won’t happen anytime soon.

2 Likes