Pre-RFC: Standardization of feature parameters

Development RFCs
1

feature: feature_parameter_names
start-date: 2024-01-10
author: Dmitry Bogatov
co-authors: (find a buddy later to help out with the RFC)
shepherd-team: (names, to be nominated and accepted by RFC steering committee)
shepherd-leader: (name to be appointed by RFC steering committee)
related-issues: (will contain links to implementation PRs)

Summary

Nixpkgs usually exports compile-time options of the upstream build system
through the named parameters of derivation function, but does it inconsistent
way.

Motivation

Developer interface for building environment with specific features enabled and
disabled is more complicated and requires more knowledge of implementation
details compared to already existing systems.

For example, Gentoo Linux has exhaustive list of all known configuration
flags and has means to
programmatically query what flags are available for particular package. Nixpkgs
has neither, and this RFC strives to rectify it.

Detailed design

  1. Derivation function MAY expose compile-time boolean options of the upstream
    build system to enable or disable a feature using parameters named either as
    enableX or withX, consistent with Autoconf naming convention.

    Autoconf ENABLE
    Autoconf WITH

  2. If compile-time feature requires build and runtime dependency on package
    libfoo, corresponding feature parameter MUST match regular expression
    ^with[^a-z]. See guidelines below for choosing name of feature parameter.

  3. If compile-time feature does not require any extra build dependencies,
    corresponding feature parameter MUST have name matching ^enable[^a-z] and
    SHOULD correspond to the upstream naming.

  4. If upstream build features and build dependencies do not map one-to-one,
    then one with feature parameter SHOULD be added for every build dependecy
    and one enable feature SHOULD be added for every upstream build feature
    intended to be exposed, and necessary assertions MUST be added.

  5. These rules are to be enforced by static code analyse linter. Since no
    static code analyzis is perfect, it shall have support for inhibiting
    warnings in individual cases that do not fit into general scheme.

  6. Parameter names matching ^(enable|with) regular expression MUST not be
    used for any other purpose. In particular, they always must be boolean.

  7. Derivation function MAY expose compile-time string or numeric options of the
    upstream build system using feature parameters that MUST match ^conf[^a-z]
    regular expression, e.g confDefaultMaildir.

  8. Due overwhelming amount of possible combinations of feature flags for some
    packages, nixpkgs maintainer is not expected to test or maintain them all,
    but SHOULD accept provided technically sound contributions related to
    configurations with non-default feature flags.

  9. Due overwhelming amount of possible combinations of feature flags for some
    packages, only configurations that has name in package set (e.g emacs-nox)
    shall be built on CI.

The migration process.

Named parameters are part of the programming interface, and renaming them is a
breaking change. As such, renaming of the parameters is done in following way:

  1. Following function is added into lib set:
let renamed = { oldName, newName, sunset, oldValue }: newValue:
   let warning = builtins.concatStringsSep " "
      [ "Feature flag"
      , oldName
      , "is renamed to"
      , newName
      , "; old name will no longer be available in nixpkgs="
      , sunset
      , "."
      ]
   in lib.warnIf (value != null) warning (if (value != null) then value else newValue);

Starting with following function:

{ lib
, stdenv
, nonCompliantFoo ? true
}:

# uses nonCompliantFoo
stdenv.mkDerivation { ... }

First step of migration is to replace it with the following:

{ lib
, stdenv
, nonCompliantFoo ? null
, enableFoo ? lib.renamed {
    oldName = "nonCompliantFoo";
    newName = "enableFoo";
    sunset = "25.11";
    value = nonCompliantFoo;
  } true
}:

# uses enableFoo
stdenv.mkDerivation { ... }

and after grace period of two releases, any mentions of nonCompliantFoo are
removed, and function becomes:

{ lib
, stdenv
, enableFoo ? true
}:

# uses enableFoo
stdenv.mkDerivation { ... }

Feature parameter naming guidelines

  1. Feature flags that require single external dependency SHOULD be named after
    that dependency. Prefix lib SHOULD be removed. For example,

    systemd => withSystemd
libgif  => withGif
curl    => withCurl

  2. When multiple feature flags require the same build dependency, for example
    derivation has optional support for FTP and HTTP protocols, any of which
    incur dependency on curl, and derivation would look like following:

    { lib, stdenv, curl, withCurl ? true, enableFTP ? true, enableHTTP ? true }:

assert withCurl -> enableFTP || enableHTTP;
assert enableFTP -> withCurl;
assert enableHTTP -> withCurl;

stdenv.mkDerivation {
   ...

   buildInputs = lib.optionals withCurl [ curl ];

   ...
}

  3. Mutually-exclusive build dependencies that provide the same feature are also
    handled with assertions. For example, if derivation has optional SSL support
    that may be provided by multiple libraries, but only one may be used and it
    must be chosen at compilation time, derivation will look like following:

    { lib, stdenv, enableSSL ? false, openssl, withOpenSSL ? false, libressl, withLibreSSL ? false }:
assert withLibreSSL -> enableSSL;
assert withOpenSSL -> enableSSL;
assert enableSSL -> withOpenSSL || withLibreSSL;
assert withOpenSSL -> !withLibreSSL;
assert withLibreSSL -> !withOpenSSL;

stdenv.mkDerivation {
   ...

   # Asserts above make sure that at most one SSL implementation will be in
   # the list.
   buildInputs = lib.optionals withLibreSSL [ libressl ]
               ++ lib.optionals withOpenSSL [ openssl ];

   ...
}

  4. When build dependency comes from particular package set, it makes set to
    name feature parameter after it. E.g build dependency on qt6.qtbase should
    have withQt6 feature parameter.

  5. Build dependency on bindings to C library SHOULD be named after underlying C library.
    For example, optional dependecy on pyqt5 Python bindings to Qt5 library should have
    withQt5 feature parameter.

Examples and Interactions

This is how one can query list of feature parameters that incur extra runtime dependencies:

$ nix eval --json --impure --expr 'with import ./. {}; builtins.attrNames gnuplot.override.__functionArgs' | jq '.[]| select(startswith("with"))'
"withCaca"
"withLua"
"withQt"
"withTeXLive"
"withWxGTK"

I picked the gnuplot as example since it is the closest to be compliant with proposed rules.

Drawbacks

  1. The migration process involves renaming feature parameters, so these changes
    will likely conflict with other outstanding changes to the derivation, and potentially
    even with automatic version bumps.

Alternatives

Group all feature parameters into separate attrset parameter

Instead of writing

{ lib, stdenv, enableFoo ? true }:

# uses enableFoo
stdenv.mkDerivation { ... }

derivation can be written as

{ lib, stdenv, features ? { enableFoo = true; }}:

# uses features.enableFoo
stdenv.mkDerivation { ... }

It is definitely looks cleaner, but unfortunately hits several limitations of
the Nix language.

  1. Nix language provides way to introspect function argument names,
    but no way to learn their default values. So this approach gives consistency, but no way to query list of derivation feature parameters.

  2. Overrides become much more verbose. Simple and ubiquitous

    bar.override { withFoo = true; }

    becomes unwieldy

    bar.override (old: old // { features = old.features // { withFoo = true }; })

    That can be simplified by introducing overrideFeatures function, but we
    already have way too many
    override functions. Also, this version will silently override nothing in case
    of typo.

Any approach that tries to pass attribute set to function will have these
issues. Using existing config.nix
is no different.

Do nothing

Avoids all the work and drawbacks, but there is no evidence that consistency
problem will solve itself evolutionary.
For some build dependencies, we have multiple equally popular feature parameter
names, and people keep picking random one when adding new packages.

Prior art

As mentioned in motivation part, the best in class of feature flag
configuration system is Gentoo Linux:

There is not much work I can find about this problem in Nixpkgs other than my
previous attempts to solve it on package-by-package basis:

Unresolved questions

This RFC makes it possible to introspect feature parameters of particular
derivation, but still does not provide simple and efficient way to list all
existing feature parameters.

Future work

There are other configuration scenarios not covered by this RFC:

  • Optional dependencies in shell wrappers (e.g passage).
  • Finding way to get list of all existing feature parameters. That can be possibly done by building and distributing the index separately,
    like nix-index does it.

Changelog

  1. Changed wording to not imply that every upstream build system knob SHOULD be
    exported via feature parameters. (Thx: @7c6f434c)

  2. Relaxed wording on the name of feature parameters to avoid painting ourselves
    into ugly and non-intuitive names. (Thx: @7c6f434c)

  3. Fix typo in regex to be consistent that feature flag name can’t have small
    letter after with|conf|enable prefix. (Ths: @don.dfh)

  4. Explicitly mention that static code analysis has support for overrides based
    on human judgement call. (Thx: @7c6f434c)

  5. Clarify solution scenarios when build inputs and feature flags don’t match
    one-to-one. (Thx: @Atemu, @7c6f434c)

  6. Refine the deprecation plan to make sure the warning includes the sunset
    timestamp. (Thx: @pbsds)

  7. Add rules about non-boolean feature parameters. (Thx: @Atemu, @pbsds)

  8. Set expectations for building and maintaining multiple configurations. (Thx: @pbsds)

9 Likes
2

Does this put a SHOULD on adding a ton of feature parameters compared with current practice where only a few significant parameters are handled?

How is this interpreted for compile-time features with multiple dependencies?

2 Likes
3

  1. Good point, thank you. That wasn’t intended to be in the scope of this RFC. Changed to MAY.

  2. I didn’t thought about it. Can you give an example of this scenario, please?

4

Roughly: you could have some cool feature, which changes the behaviour and is semantically an enable flag, but oh also it needs to regenerate some parser so depends on flex and bison.

Or maybe you have enableStandardGraphicsFormats which expects both libpng and libjpeg because absolute-minimum-pnm-only makes some sense in rare situations but supporting arbitrary combinations is not worth the work.

5

  1. flex and bison are nativeBuildInputs, so this is just enabelCoolFeature. You think I should add this case to examples?

  2. Let’s do another example, also convoluted. Pdf reader with optional support of opening https:// URI and optional support of djvu, but these two features have one shared --kitchen-sink flag. I would call it undefined behavior and defer to the judgement of individual Nixpkgs maintainer.

6

How does this account for non-Boolean build-time parameters? I can imagine there may exist things like someValue ? "foo".

1 Like
8

No rulings about this. Question is listed in “Future Works” section.

9

This does create some inconsistency because if you only need one package for GUI you must call the flag withPyQt or whatever, but once you have two dependencies, you probably end up with enableGui

10

How about something along the lines:

If single upstream build system features requires multiple build dependencies,
the feature parameter name MUST be named after one of them. Which exactly is left at discretion of individual nixpkgs maintainer,

11

Maybe I should actually look at what happens in Nixpkgs

ffmpeg/generic.nix

optionals (withNvdec || withNvenc) [ (if (lib.versionAtLeast version "6") then nv-codec-headers-11 else nv-codec-headers) ]

(the --enable- flags are different though)

The functionality provided here has no legal implementation under your wording.

12

Yeah, this is complicated. My intended scope is humble – get one canonical naming for clear as day scenarios. How about I move that out of scope too?

If single upstream build system features requires multiple build dependencies,
the feature parameter name SHOULD be named after one of them. Which exactly is left at discretion of individual nixpkgs maintainer. This RFC makes no ruling in case when
multiple upstream feature flags require the same build dependency.

13

buildInputs = lib.optionals enableGUI [ (if stdenv.isLinux then qt6.qtwayland else qt6.qtbase) ];

It is not allowed to call this withQt ?

(For the record, there are cases where enableGUI honestly seems to be a better choice than withQt/withGtk3…)

14

Where this is from? Because it looks to me as quite clean case of withQt6.

15

How do you put the definition for static analysis to accept that, and will it break if the things from the qt6 package set are inherited into the scope?

(Is winePackages.stable still allowed to correspond to withWine by the way?)

(I grepped for a specific bad pattern, some minor media player, IIRC, if it is relevant, I can grep again)

16

I think no tool can hope to handle all scenarios, so it will have support for I am human, I know better overrides, which would attract extra attention in the review process. This is how it is practiced for long time by Debian project.

Lintian

By the way, no override would be necessary in your qt6 example, it fits perfectly.

{ lib, stdenv, qt6, withQt6 }:

stdenv.mkDerivation {
  buildInputs = [
   (if stdenv.isLinux then qt6.wayland else qt6.base)
  ]
}

Wine will need an override. Or actually, no, in addition of stripping lib prefix I will add clause about stripping Packages suffix.

17

Love the idea, Gentoo USE flags was one of my favorite features :blush:

Small nitpick (in hope I didnt mis-parse the regex)
You require camel casing in 2 for with explicitly, but not in 3 for the enable definition - the examples for it are written in camel casing though.

1 Like
18

In this case, I think

needs to mention the overrides. We have precedents where absolutely unusable checks have been introduced with no overrides, like black linting of NixOS tests (it had line length issues with string interpolation), so I want to be safe here.

Should the final numbers be optionally strippable? There are cases where the user has a choice of Gtk2 or Gtk3, but sometimes withQt4 → withQt5 → withQt6 upgrades happen without any choices or reasons to make the end-user to update configuration.

[pythonPackages.pyqt] is still withPyqt, right? (Do humans have the right to use withPyQt without full override?)

19

I think a better solution would be to have both; enable and with. The enable would be set depending on its with dependencies.

If you set withFlex or withBison false, the feature would simply be disabled by default. It’d be expressed like this withFooFeature ? withFlex && withBison. (Assuming controlling these two would actually be of interest.)

Forcibly enabling withFooFeature here would have to be defined as undefined behaviour.

20

I’ll apply all comments to the document and will post full revised edition. probably today evening. I’ll add mention of overrides.

I agree that withPyqt is ugly, but ugly is in the eye of beholder, and allowing beauty to get in the way of uniformity is exactly purpose of this RFC.

21

Flex and bison are bad examples. They are native build inputs, they do not need withFlex parameters.