The papercut thread - post your small annoyances/confusions here

I think it is sufficient! I think we should heavily document this; it seems very useful!!

Example usage that I am now using in a module I’m building

{ lib, config, ...}: 
let
  inherit (lib) mkIf mkMerge mkOption mkEnableOption ;
  inherit (lib.types) attrsOf str;
in
{
  options.services.journal-upload = {
    enable = mkEnableOption "journal-upload";
    config = mkOption {
      type = attrsOf str;
    };

  };
  options.services.journal-remote = {
    enable = mkEnableOption "journal-remote";
    config = mkOption {
      type = attrsOf str;
    };
  };

  config = mkMerge [

    (mkIf config.journal-upload.enable {
      systemd.sockets.systemd-journal-upload.wantedBy = [ "sockets.target"];
      environment.etc."journal-upload.conf".text = "";
    })

    (mkIf config.journal-remote.enable {
      systemd.sockets.systemd-journal-remote.wantedBy = [ "sockets.target"];
      environment.etc."journal-remote.conf".text = "";
    })
  ];
  
}


10 Likes

I have a lot of trouble finding the definitions of particular functions.

I have a lot of trouble with having to cleanroom a compiler toolchain (Rust in this case) when the version I want to use isn’t in nixpkgs.

Several compiler toolchains (again, Rust) depend on the auto-updater (rustup) instead of on a precise .tar.gz.

(I know I should really get involved with the Rust derivation maintenance, or at least talk to the people involved to understand the philosophy.)

4 Likes

Yeah, that explanation is really very internals-focused. Here’s a (hopefully) more comprehensible explanation that focuses more on how to use it, and only on implementation details that directly affect the use of it:

Putting a dependency a in the propagatedBuildInputs of derivation b has two main effects:

  1. It treats a as a dependecy of b, just as if you had buildInputs = [ a ]; in b
  2. It “propagates” this dependency relationship onwards to all derivations that depend (directly or transitively) on b

For example, if you have a derivation c with b specified as a buildInput, the underlying nixpkgs machinery will ensure that c also effectively has a as a propagatedBuildInput, meaning it gets a as a direct dependency (available in its builder) rather than simply depending on it transitively (as a result of b being built with it).

As a is effectively added to c’s propagatedBuildInputs, anything that depends on c will also get a added to their own propagatedBuildInputs. The overall effect is that anything that has b anywhere in its dependency closure gets a direct dependency on a (at least if we ignore cross-compilation cases).

This effect is used in several ways in nixpkgs and NixOS. A few of the language-specific infrastructures make use of it to ensure that executables have access to their run-time dependencies.

For example: Python library/module dependencies are specified as propagatedBuildInputs, and the Python mkDerivation wrapper (buildPythonPackage) wraps any Python executable scripts produced by it to give them references to the paths of all their Python dependencies. This ensures that Python scripts have access to their full (Python) dependency closure at run-time, which is necessary because it is an interpreted language.

For the languages-specific infrastructures that do this, propagatedBuildInputs can also be seen as “a list of dependencies that need to be available to $lang scripts at run-time”.

17 Likes

Thanks, this did indeed solve my problem!

@Shados This is a really great explanation. Thanks for taking the time to write this!

  1. It “propagates” this dependency relationship onwards to all derivations that depend (directly or transitively) on a

I guess this should be “derivations that depend (directly or transitively) on b”?

any propagatedBuildInputs in the dependency closure of a profile will also find their way into the profile’s paths.

The net effect is that propagatedBuildInputs can also be seen as a list of “dependencies that need to be available at run-time”.

I think this is the part I was really missing! This makes total sense, and I can see how useful it could be when writing derivations!

In general nix requires a lot of upfront investment to be efficient with it. Unlike guix a lot of things like generating arbitrary containers are relegated to 3rd party projects. For example using cntr to debug derivations is great but I’m not using it because it’s relatively hidden and forgot how to use it.

Some good old ones:

  • multiple installation instructions: nix-env -iA nixpkgs.hello vs nix-env -iA nixos.hello
  • too many channels; I think it’s confusing to have a nixos-* and nixpkgs-* channels.
  • channels don’t map to hydra jobset names. Eg nixpkgs-unstable is nixpkgs/trunk on Hydra.
  • nix-collect-garbage -d doesn’t clean the old /boot entries. This affects everyone using NixOS.
13 Likes

Are you sure about this? I thought we needed to use propagatedUserEnvPkgs for that, otherwise I do not see why would it exist.

For Python libraries, I think they are included in the runtime closure by buildPythonApplication, not that they are picked up from profile.

Edit: In my mental model, the profiles are managed by Nixʼs buildEnv which is different from Nixpkgsʼs so I do not see how it would know about Nix specific features like propagated build inputs.

Edit2: Thinking about that, Nix would have to know about propagatedUserEnvPkgs as well. And looking at the source code, it seems to. I see no such case for propagatedBuildInputs.

Edit3: Really, being a derivation is not anything magical. Rather Nixʼs derivation is rather useless by itself. There is a lot of orchestration done by the generic builder used by default by stdenv.mkGeneration which adds lot of data used by other such derivations instrumented by it. Look at Nix pills to learn how it works.

1 Like

For what it’s worth, specifically for Rust there is the Mozilla overlay, which is (more or less) feature-equivalent to rustup where Rust tooling is concerned. It does unfortunately also suffer from a bit of a lack of documentation.

I have never heard of this. Where do I find more information about it?

1 Like
3 Likes

I do need to try that. Do you know who would be the proper folks for me to contact? I keep wanting to contribute to nixpkgs, but I’m really intimidated.

1 Like

As a preface to my comments, I really like NixOS and have championed it with others. But there are some things that could use some devoted attention:

  • nix-env -qaP is too slow. There are other tools being proposed but none of them are canonical and promoted in the manual.
  • A more complex search capability for packages and options is needed. Many searches return too many results to sort through efficiently. This is especially important given that the packages/options documentation are huge lists without any structure.
  • I’ve used NixOS for a couple of years but I am still tripped up by the sub-categories, like vimPlugins, gnome{2,3}, python{27,37}Packages, etc. Having hierarchical namespaces isn’t bad but the tools/docs for easily using them are lacking. (It should be possible to automatically generate a drop-down menu of the sub-spaces to support more sophisticated searches for example.)
  • Configuring emacs and extensions is still a pain and I can’t figure out how to do it well. (There are other packages as well.)
  • Documentation, documentation, documentation! :slight_smile: It seems like there are docs for getting started and docs supporting “power users” but very little in the middle. Especially for those of us who like and use NixOS but for whom our day job is something else so we won’t become power users anytime soon (without help).
6 Likes

I may not be your best offer, but you are much welcome to contact
me :-).

2 Likes

Sometimes you can use builtins.unsafeGetAttrPos in the REPL to find them, though often what you actually find is a trivial wrapper.

Ah, that’s true. I think this is at least the third time I’ve forgotten propagatedUserEnvPkgs exists :joy:, mostly because it’s undocumented (another papercut for the list!). I’ll amend my previous post in the morning.

2 Likes

The reason it is undocumented is because it is frowned upon. After all, it breaks purity.

That seems like a bad reason not to document it. All that accomplishes is making it less obvious why it’s a bad idea, because now people just discover its existence and go “hey, that’s handy!” without understanding the reasons why you might want to avoid it…

People will eventually find anything that exists anyway. In the official documentation you want make as sure as possible that they find it on your terms, and correctly understand what it does and what its consequences are.

7 Likes

@deliciouslytyped : “Cannot coerce set to string” is my arch nemesis.

@Shados : Thank you for demystifying propagation!
Would you mind adding this to the manual; or else would you object to me adding it?


I have a paper cut:

Many super useful functions are defined throughout Nixpkgs inside of let ... in blocks. This is great for functions that truly need to be kept local for a particular file; but often the ability to import and re-use these functions would be super convenient.
Often these functions are specific to a collection of packages so they don’t make it into lib, which makes sense; but something like “exports” in a file or at the very least a migration of many functions into lib.nix in many directories would be awesome. A prime example is “documentation” which has loads of useful functions that I often copy and paste parts of into peojects to collect info about my system. I have a post called “nix fuzzy search” or something along those lines on this board that demonstrates this.

I would be more than happy to implement “exports” or start migrating things into lib.nix if there is supporting consensus.


Another “hot take”:

Documentation for a subsection of Nixpkgs should consistently appear in a folder’s README.md. The Nixpkgs/NixOS manual should be built by sourcing all of or sections of these files.

I have been using Nix for well over a year and frankly I have never added to our documentation because it feels like this “other thing” that I need to learn how to use. I honestly feel like from a “design/contribution experience” perspective Nix Manuals feels divorced from the actual packages and tools. If I’m hacking around in a folder, I expect a README.md to be nearby for me to reference or add things to as I learn and contribute. If I’m being perfectly honest I couldn’t tell you off the top of my head where the documentation files are kept in the repo, but can easily recall where many packages and modules reside. I may be alone in this regard, but if I’m not, this might help explain why we often end up with outdated or non-existant documentation.

JFC I know that these files often exist. On that note these files are sometimes more up to date than our Manuals; sourcing them into the Manual’s build rather than having “two versions” seems reasonable. I think that altering the pattern for how we build the manual and write documentation, to make it more approachable and consistent could help to solve some of the documentation issues that we face.

Imagine this radical scenario if you will:
You open Nixpkgs on GitHub’s webpage and dive into the haskell-modules directory; magically a README.md appear as nicely formatted HTML in your browser :face_with_hand_over_mouth:. You open the lib directory and GitHub serves up some beautifully highlighted usage examples, and helpful tips left by contributors of days past :raised_hands:t3:. Before I lose you in this fantasy lets recollect.

I often make quick tweaks to other repo’s docs directly through GitHub’s web UI when I’m browsing, and it would be a delight to have that same convenience in our repo. If we want better documentation, I suggest we begin by making it more convenient to contribute documentation.


Follow up:
I started digging into the documentation build process and discovered that it contains/builds some fantastic Markdown documents for various subsections of the repo, that would be perfect to put into the relevant parts of the repository! Currently the process (for some parts) is : (XML DocBook ↔ Markdown) → HTML. I am basically suggesting that we instead do : Markdown → XML DocBook → HTML. The repo has both Markdown and XML now in some cases that hold identical content.
This process/pattern does not exist for all parts of the manual, and it’s not immediately clear to me why it bothers building the markdown files at all honestly since it’s not as if most people are ever discover them. You can find the existing ones at docs/language-frameworks/*.section.md and docs/introduction.chapter.md. I want to generate these for the remaining sections of the manual, disperse them to their relevant directories, and then alter the manual build script to perform the aforementioned Markdown → XML transformation.
Edit: I had initially thought that the Markdowns were generated from XML, but after looking back at a fresh version of the repo I realized that both versions exist, holding identical contents.


Edits: Merged my multiple comments, and “Follow Up”.

10 Likes

I just keep not having time to work on this but this might help with the lib stuff, though I think a possibly less complicated but less flexible solution would be to just use more rec attrsets instead of let expressions. I don’t know if this has any drawbacks. I don’t know any lisp but maybe letrec is analogous.

Namely: GitHub - deliciouslytyped/nix-rootedoverlay: A simple Ouroboros is overrated. and an example at GitHub - deliciouslytyped/nix-ghidra-wip: WIP repository for prototyping ghidra packaging based on deliciouslytyped/nix-rootedoverlays . I really hope to getting around to cleaning this up some more and getting some pro community feedback soon…

2 Likes

To me, that sounds quite similar to “security through obscurity”, and generally like a poor reason to leave something undocumented. I think it is better to document exactly the kind of situations in which it can be used (because it must).

What do you mean when you say it “breaks purity”? The output of a profile with packages that use it is still defined solely by its inputs, yes…?

Sure, go ahead.

3 Likes