The papercut thread - post your small annoyances/confusions here

Shados · June 9, 2019, 2:32pm

Ah, that’s true. I think this is at least the third time I’ve forgotten propagatedUserEnvPkgs exists , mostly because it’s undocumented (another papercut for the list!). I’ll amend my previous post in the morning.

jtojnar · June 9, 2019, 2:44pm

The reason it is undocumented is because it is frowned upon. After all, it breaks purity.

joepie91 · June 9, 2019, 2:57pm

That seems like a bad reason not to document it. All that accomplishes is making it less obvious why it’s a bad idea, because now people just discover its existence and go “hey, that’s handy!” without understanding the reasons why you might want to avoid it…

People will eventually find anything that exists anyway. In the official documentation you want make as sure as possible that they find it on your terms, and correctly understand what it does and what its consequences are.

Growpotkin · June 9, 2019, 7:13pm

@deliciouslytyped : “Cannot coerce set to string” is my arch nemesis.

@Shados : Thank you for demystifying propagation!
Would you mind adding this to the manual; or else would you object to me adding it?

I have a paper cut:

Many super useful functions are defined throughout Nixpkgs inside of let ... in blocks. This is great for functions that truly need to be kept local for a particular file; but often the ability to import and re-use these functions would be super convenient.
Often these functions are specific to a collection of packages so they don’t make it into lib, which makes sense; but something like “exports” in a file or at the very least a migration of many functions into lib.nix in many directories would be awesome. A prime example is “documentation” which has loads of useful functions that I often copy and paste parts of into peojects to collect info about my system. I have a post called “nix fuzzy search” or something along those lines on this board that demonstrates this.

I would be more than happy to implement “exports” or start migrating things into lib.nix if there is supporting consensus.

Another “hot take”:

Documentation for a subsection of Nixpkgs should consistently appear in a folder’s README.md. The Nixpkgs/NixOS manual should be built by sourcing all of or sections of these files.

I have been using Nix for well over a year and frankly I have never added to our documentation because it feels like this “other thing” that I need to learn how to use. I honestly feel like from a “design/contribution experience” perspective Nix Manuals feels divorced from the actual packages and tools. If I’m hacking around in a folder, I expect a README.md to be nearby for me to reference or add things to as I learn and contribute. If I’m being perfectly honest I couldn’t tell you off the top of my head where the documentation files are kept in the repo, but can easily recall where many packages and modules reside. I may be alone in this regard, but if I’m not, this might help explain why we often end up with outdated or non-existant documentation.

JFC I know that these files often exist. On that note these files are sometimes more up to date than our Manuals; sourcing them into the Manual’s build rather than having “two versions” seems reasonable. I think that altering the pattern for how we build the manual and write documentation, to make it more approachable and consistent could help to solve some of the documentation issues that we face.

Imagine this radical scenario if you will:
You open Nixpkgs on GitHub’s webpage and dive into the haskell-modules directory; magically a README.md appear as nicely formatted HTML in your browser . You open the lib directory and GitHub serves up some beautifully highlighted usage examples, and helpful tips left by contributors of days past . Before I lose you in this fantasy lets recollect.

I often make quick tweaks to other repo’s docs directly through GitHub’s web UI when I’m browsing, and it would be a delight to have that same convenience in our repo. If we want better documentation, I suggest we begin by making it more convenient to contribute documentation.

Follow up:
I started digging into the documentation build process and discovered that it contains/builds some fantastic Markdown documents for various subsections of the repo, that would be perfect to put into the relevant parts of the repository! Currently the process (for some parts) is : (XML DocBook ↔ Markdown) → HTML. I am basically suggesting that we instead do : Markdown → XML DocBook → HTML. The repo has both Markdown and XML now in some cases that hold identical content.
This process/pattern does not exist for all parts of the manual, and it’s not immediately clear to me why it bothers building the markdown files at all honestly since it’s not as if most people are ever discover them. You can find the existing ones at docs/language-frameworks/*.section.md and docs/introduction.chapter.md. I want to generate these for the remaining sections of the manual, disperse them to their relevant directories, and then alter the manual build script to perform the aforementioned Markdown → XML transformation.
Edit: I had initially thought that the Markdowns were generated from XML, but after looking back at a fresh version of the repo I realized that both versions exist, holding identical contents.

Edits: Merged my multiple comments, and “Follow Up”.

deliciouslytyped · June 9, 2019, 8:46pm

I just keep not having time to work on this but this might help with the lib stuff, though I think a possibly less complicated but less flexible solution would be to just use more rec attrsets instead of let expressions. I don’t know if this has any drawbacks. I don’t know any lisp but maybe letrec is analogous.

Namely: GitHub - deliciouslytyped/nix-rootedoverlay: A simple Ouroboros is overrated. and an example at GitHub - deliciouslytyped/nix-ghidra-wip: WIP repository for prototyping ghidra packaging based on deliciouslytyped/nix-rootedoverlays . I really hope to getting around to cleaning this up some more and getting some pro community feedback soon…

Shados · June 10, 2019, 1:41am

To me, that sounds quite similar to “security through obscurity”, and generally like a poor reason to leave something undocumented. I think it is better to document exactly the kind of situations in which it can be used (because it must).

What do you mean when you say it “breaks purity”? The output of a profile with packages that use it is still defined solely by its inputs, yes…?

Sure, go ahead.

jtojnar · June 10, 2019, 8:53am

Right. Technically, the nix-support/propagated-user-env-pkgs side channel is part of the built derivation so perhaps I should have called it breaking encapsulation. I am looking at the mechanism as an side effect, as it allows arbitrary dependency to squeeze itself into the profile’s output ~~without the knowledge or control of the dependents.~~ (Edit: apparently, it only takes effect when in a package user installs, not in transitive dependencies; unless you are QT, and we all know the issues this causes.)

The encapsulation is an important property of Nix that allows us to have multiple programs whose dependencies would conflict on traditional distros installed at the same time. If the dependencies leak into the profile they can collide with other leaked dependencies or even user’s own programs. What is even worse, this leads to a disparity between programs that are installed and those that are run, for example, from the result/ directory.

Finally, it can also lead to confusion about why certain packages are in the profile.

joepie91 · June 10, 2019, 11:27am

Another debugging papercut:

Stacktraces are basically useless. They’re in the inverse order of what you would expect (coming from most other programming languages) and this is not clearly indicated, and because they are not aligned in any way, it is difficult to scan for the thing you’re looking for. You’ll also often get illogical stacktraces due to lazy evaluation, and there seem to be no internal execution tracking features (that eg. remember the origin of a particular value) to compensate for this.

Edit: A concrete example I ran into yesterday… when using buildFHSUserEnv, it has an attribute named targetPkgs that expects a function which returns a list. When you accidentally pass in a list directly, you get this stacktrace:

error: while evaluating the attribute 'buildCommand' of the derivation 'sidequest-0.3.1' at /home/sven/projects/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating the attribute 'text' of the derivation 'sidequest-0.3.1-init' at /home/sven/projects/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14:
while evaluating the attribute 'buildCommand' of the derivation 'sidequest-0.3.1-fhs' at /home/sven/projects/nixpkgs/pkgs/build-support/build-fhs-userenv/env.nix:190:3:
while evaluating the attribute 'passAsFile' of the derivation 'sidequest-0.3.1-usr-target' at /home/sven/projects/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14:
attempt to call something which is not a function but a list, at /home/sven/projects/nixpkgs/pkgs/build-support/build-fhs-userenv/env.nix:31:17

… which does not mention targetPkgs anywhere, nor contains any stacktrace item that points at a line in the offending file, at all. It all just points at nixpkgs utility things that have no apparent relation to targetPkgs.

thomas · June 10, 2019, 11:36am

I’d like to echo what was said over relating to setting up build/dev environments, especially in especially using nix-shells. I’ve been trying to do it for Rust, Python, Haskell, Elm, ReasonML ++, and every time it’s taken me a good while to figure out what I need and how I go about it. It’s easier for languages using nothing but the standard mkDerivation, but when you need to start locking things down to a specific compiler etc., it gets hairy.

I think this is primarily due to a lack of documentation for language-specific derivations (Haskell, Python). The manual has some info, but I still found it hard to work with.

For instance, I wanted to create a couple small python scripts that would act on files and have them available in my PATH in a repository, but I couldn’t find out how to build it into a package /script in the end, so I’m still just using the full path every time I’m calling them.

So maybe what I want is some more documentation on nix-shell in general, past what you get in the nix-pills. Not to mention some documentation on all the options (buildInputs, propagatedBuildInputs, etc) in mkDerivation: what they are and what they do. (Thanks for the explanation on some of that higher up!)

joepie91 · June 10, 2019, 10:28pm

Upgrading NixOS involves fixing the deprecated/changed/renamed/etc. things one by one, which is an arduous process. There ought to be a tool that can a) automatically change things that don’t require human supervision (eg. simple renames), and b) give you an immediate overview, at once, of everything that needs changing for an upgrade to a new NixOS version, so you can fix/restructure it all at once.

joepie91 · June 11, 2019, 3:02pm

For some services, a system rebuild (that upgrades or installs the service’s packages) may result in the service failing to start the first time, making the entire system rebuild fail. A second rebuild of the exact same configuration will often start the service just fine. Presumably there’s some state somewhere that makes this non-deterministic.

deliciouslytyped · June 11, 2019, 3:18pm

I’ve had similar but different (I think? It’s been a while) problems with tinc a while back, with service orders and I…don’t know. Stateful state is stateful. Made it real hard to get a reliable vpn while being able to do rebuilds…

Oh, oh, oH- that reminds me. We still don’t have a built in automatic rollback on network failure mechanism?

joepie91 · June 11, 2019, 3:20pm

Yeah, I believe that tinc is one of the services where I’ve had this happen as well. I recall it requiring a bunch of stateful setup ahead of time to make it work reliably.

Edit: Just did a fresh install with an existing fairly complex configuration, and the “CUPS Scheduler” service seemed to suffer from the same issue of having to rebuild twice before the service would start correctly.

joepie91 · June 11, 2019, 8:06pm

More documentation papercuts:

The pname attribute (in mkDerivation) is not documented anywhere, in particular how it differs from name. Apparently the difference is that name is automagically composed of pname + version.
The makeDesktopItem utility function, used for creating menu entries and such, does not seem to be documented anywhere either.

teto · June 12, 2019, 3:14am

the papercuts topic seems to have evolved into an “axecuts” topic.

I just wish we had “man nix”

joepie91 · June 12, 2019, 1:36pm

Just reminded of this one by a question on IRC:

There’s no way to run multiple instances (with a different configuration) of the same NixOS/nixpkgs service, unless the service is specifically designed to allow that. This conflicts pretty harshly with the “install any amount of copies of the same package” philosophy, and poses issues for more complex setups where you’d want to eg. run different versions of the same DB with a different ‘backing package’, on a different port.

Melkor333 · June 12, 2019, 8:26pm

Current package states and open issues to a package seem to be often unknown or ignored.

It happened twice to me that somebody pushed a breaking commit directly to the master and while figuring that out I came along some other PR which was kept back. The PR contained the same thing as the breaking commit and was kept back because it was known that it would break things… One time the PR was kept back for over 2 months when the commit was pushed.

I love to use NixOS unstable and report bugs, but I often find that people don’t seem to really know what’s going on with certain packages/issues and mis(sing )communication or duplicated work are not as rare as they maybe should be. I don’t want to say people have to know about things (Nixpkgs is too huge) but at least to me it seems that such package orientated information is harder to get than it should be.

Shados · June 13, 2019, 7:00am

Sure there is, it’s “just” non-obvious and annoyingly indirect:

containers = {
  postgres_one = { 
    config = { config, pkgs, ... }: {
      services.postgresql = {
        enable = true;
        package = pkgs.postgresql_9_6;
    };
    forwardPorts = [
      { hostPort = 15432; containerPort = 5432; protocol = "tcp"; }
    ];
  };
  postgres_two = { 
    config = { config, pkgs, ... }: {
      services.postgresql = {
        enable = true;
        package = pkgs.postgresql_10;
    };
    forwardPorts = [
      { hostPort = 15433; containerPort = 5432; protocol = "tcp"; }
    ];
  };
};

Here’s a papercut I ran into today, namely: @-patterns do not capture formal argument default values.

Take this test.nix:

with builtins;
{
  a = let
    f = { foo ? "bar", ... }@args: args;
  in f {};
  b = let
    f = { foo ? "bar", ... }@args: args // { inherit foo; };
  in f {};
  c = let
    f = { foo ? "bar", ... }@args: args // { foo = args.foo; };
  in f {};

  # Which makes it tricky to generically pass on @args to another function, inclusive of defaults:
  d = let
    f = { foo ? "bar", ... }@args: g args;
    g = attrs: attrs;
  in f {};

  # Although you can do something awkward like:
  e = let
    f = { ... }@args': let
      defaults = { foo = "bar"; };
      args = defaults // args';
    in g args;
    g = attrs: attrs;
  in {
    ea = f {};
    eb = f { foo = "nak"; };
  };
}

And then:

$ nix-instantiate --strict --eval test.nix -A a
{ }
$ nix-instantiate --strict --eval test.nix -A ab
error: attribute 'ab' in selection path 'ab' not found
$ nix-instantiate --strict --eval test.nix -A a
{ }
$ nix-instantiate --strict --eval test.nix -A b
{ foo = "bar"; }
$ nix-instantiate --strict --eval test.nix -A c
error: attribute 'foo' missing, at ~/tmp/nix/defaultargs/test.nix:10:52
(use '--show-trace' to show detailed location information)
$ nix-instantiate --strict --eval test.nix -A d
{ }
$ nix-instantiate --strict --eval test.nix -A e
{ ea = { foo = "bar"; }; eb = { foo = "nak"; }; }

volth · June 13, 2019, 7:06am

Please use lib.bitAnd, lib.bitOr and lib.bitXor
There are lib.bitNot (which implemented in pure Nix and that’s why it is not in builtins. ) and fallback implementation of all the functions for older Nix.
builtins.bitAnd … work since a particular version of Nix (2.2.x ?) while lib.bitAnd should work even in Nix1, using builtins.bitAnd when available

github.com

NixOS/nixpkgs/blob/6cf583cf2f8d3cb89fa8a4d6edd86f4236862ea6/lib/trivial.nix#L67


      
          bitOr = builtins.bitOr
            or (import ./zip-int-bits.nix
                (a: b: if a==1 || b==1 then 1 else 0));
          
          /* bitwise “xor” */
          bitXor = builtins.bitXor
            or (import ./zip-int-bits.nix
                (a: b: if a!=b then 1 else 0));
          
          /* bitwise “not” */
          bitNot = builtins.sub (-1);
          
          /* Convert a boolean to a string.
          
             This function uses the strings "true" and "false" to represent
             boolean values. Calling `toString` on a bool instead returns "1"
             and "" (sic!).
          
             Type: boolToString :: bool -> string
          */
          boolToString = b: if b then "true" else "false";

Zimm_i48 · June 13, 2019, 8:07am

src=./. in a default.nix will copy to the store with a name that depends on the name of the folder you are in, which is really a shame for reusability of binary builds.