Struggling with NixOS: Can't figure out how to get some apps to work without crude hacks

Help
1

I’ve been struggling with NixOS and these bug reports have some of the gory details:

Pretty much all the problems I’ve had involve apps looking for resources (e.g., typelib files, schemas) in some path, not being able to find them, and then breaking in seemingly inscrutable ways, either involving an error message to decipher or, worse, a crash. They’re problems that by and large disappear when the FHS is used.

I’ll lay out some of the details of the problems, and then ask about solutions.

With the Workrave package, the problem is twofold. First, it has a GNOME extension that isn’t patched to point to the correct typelib files, and even the patched extension will crash the GNOME shell unless XDG_DATA_DIRS contains a path for Workrave’s gsettings schema files.

The other big problem I’ve had has been Python scripts that use D-Bus via either the pydbus or dasbus modules. (Those scripts are used with Workrave, FWIW). So far, the way I’ve gotten the scripts to work is to run them like this:

GI_TYPELIB_PATH=/nix/store/bkpj51fz88rbyjd60i6lrp0xdax1b24g-glib-2.84.1/lib/girepository-1.0 my_script.sh

Obviously, that’s a temporary hack meant as a proof of concept. FYI, I’ve installed the pydbus and dasbus packages by adding

(python3.withPackages (python-pkgs: with python-pkgs; [
  pydbus
  ]))

to environment.systemPackages.

I’m hoping that there has to be some sane, “canonical” way to get the environments straightened out, but I don’t know what it is. What’s the smart way to set XDG_DATA_DIRS to include the correct directories? How to get pydbus and dasbus to find where Glib’s typelib files are located? Is the situation quite as dire as described at this post?:

I would hope that this isn’t the whole story, because it would imply that NixOS is designed to break certain apps so that they can’t provide their desired functionality without clumsy interventions.

Or is there a general Nix way of getting the environments set appropriately, so I can avoid frustrations like this in the future?

2

It’s not “dire”, it’s called specifying dependencies properly… the thing that nix was designed to make happen. And if you read the whole quote, they linked to the exact section of the manual that you need to read; I’ll do you one better and link the exact subsection.

3

Notice the part that reads, “Of course, this does not work for programs that expect to see every schema installed on the system (e.g. dconf-editor). But since Nix has no concept that would match “installed” of traditional package managers that cannot really be implemented in the first place”

That part is dire, because it means that dconf-editor won’t behave as intended under NixOS.

[ETA: Also, I did “read the whole quote”. That’s how I know that the subsection to which you linked doesn’t apply to cases like dconf-editor, and it’s also a dicey solution for single-file Python scripts. That subsection is also unhelpful for cases where a Python module is imported in a REPL for testing purposes.]

4

NixOS avoids runtime resolution of dependencies in favor of build-time resolution whenever possible. That means we also avoid global search paths. This is very much a design goal, not just an accident. It allows programs with incompatible dependencies to coexist without problems on the same system. This is why python3.withPackages exists, rather than just letting you install python3 and some python packages side by side and having it automatically find them.

This also means foundational assumptions of some programs break, particularly those that fundamentally rely on that non-existant global search path (dconf-editor, as noted, or even just ldd).

The solution when something (that isn’t conceptually broken like this) can’t find what it needs is almost always to package it. Give whatever you need to have access to these dependencies a proper environment through the nix packaging machinery.

3 Likes
5

What to do for cases where creating a full package derivation is impractical? Doing a full package of every little Python script I may write with pydbus or dasbus seems like overkill, not to mention a time sink.

(Plus, there’s the not-so-small matter of finding what dependencies are even needed, because I only find out when something doesn’t work and I have to debug why.)

Also, what to do with something that is “conceptually broken” as far as NixOS is concerned, especially when that something is already packaged?

1 Like
6

I’m trying to word this carefully, but to be clear I am looking our for your best interests here. If doing things the way @tejing is suggesting seems unappealing, then you are not going to enjoy using nix/nixos, and that’s ok.

Nix is all about packaging everything and once you get the hang of it, banging out a derivation to wrap some script is not a significant time sink. OTOH, if not using nix leaves you with zero dependency problems to solve and all you’ve been burdened with by using nix is extra work, why bother? Just use something else more suited to your workflow and needs.

Everyone here obviously thinks nix pays for itself, otherwise we wouldn’t be using it. That doesn’t make this community correct of course :slight_smile:

3 Likes
7

Besides just accepting the arguable overhead as @bme suggests, you can rework your workflow such that it is not impractical.

Write fewer small scripts, and instead combine them into a conglomarate of utilities. Use code written by other people instead of your own hacks. Try out ecosystems less married to the concept of “installation”. Or write your own alternatives that fit more sanely into the ecosystem you are using (dbus schemas are not at all necessary to make use of the dbus, they’re basically just documentation).

Imagine the pain if you didn’t do that work and it suddenly stopped working just because you installed a random package with a schema that happens to share a name with yours. Oh, wait, that’s how it works on traditional distros.

NixOS means doing work upfront to ensure things stay functional. The FHS is doing things ass-backwards; Taking away upfront work to actually resolve what’s going on and leaving it to random failure and debugging - usually by other people, or at least a version of you with much less fresh knowledge and far less time - later.

This is very abstract; Depends on what we’re talking about. Usually there are escape hatches, and that thing is just waiting for someone to properly implement those in a module.

Other times the package just exists for whatever functionality it exposes without proper runtime resolution. In those cases, you can attempt working upstream to fix it. Or you may need to be the one developing an alternative if upstream is uncooperative.

NixOS is part of a larger paradigm shift towards more maintainable software. Some things will be left behind and that’s ok.

You’re seeing some resistance here, because it’s kind of like running into the Rust forums and complaining about the lack of an Any type.

I can imagine a young Python dev doing that, but I think anyone who’s spent a few minutes reading the Rust book would understand how silly the suggestion even is.

Complaining about the lack of global search paths (even in pursuit of a specific thing) on NixOS is similar, even if the ecosystem is less mainstream and it may therefore seem more reasonable to question fundamental design goals.

2 Likes
8

Have you done that yourself? And do you have an example?

9

I maintain around 40kloc of nix as part of my day job. We have a couple of non-trivial python applications running under gui wrapper hooks, but it’s not my property so I can’t post them here, unfortunately.

I realise this is going to seem like a cop out and I have no way of proving what I said is true, so take it for what you will. On my own systems I don’t run gnome, and I don’t use python unless someone is paying me.

If you are willing to accept other examples of nixing everything I’ll see what I can dig out that might be helpful.

10

Here’s some more or less random Django plugin that happened not to be in Nixpkgs yet. It may look like a lot of code, but it’s essentially filling a form. All the others look the same. (This particular example isn’t even en par with Nixpkgs standards, so don’t take it for best practices.) Since all contributors will need to consciously do to work with your code is install Nix, as long as your CI stays green, your project and development environment is exactly one command away regardless of complexity, and you’ll effectively never have to think about that thing again. For me this is worth every minute learning Nix stuff.

1 Like
11

Actually, that looks more like a situation that could happen if I was careless in installing stuff from the AUR. :grimacing: Heh.

I think that’s half true. The other half is that when things break on NixOS due to a lack of global runtime information, it isn’t obvious that the problem is missing runtime information. At first, it just looks like random breakage.

In some cases, that’s because a package is packaged incorrectly. That seems to be the case for the Workrave package, which does come with a GNOME extension – but one that isn’t patched for use with NixOS. Looks like a partial solution to that is at “Gnome extention Hanabi on nixos - #12 by jtojnar”, and if that works with my hacked version of the Workrave extension, that would obviate a need to mess with XDG_DATA_DIRS, and I can add that information to my bug report for the Workrave package.

In other cases, the underlying problem is that some piece of software can’t be patched or wrapped to transparently work with NixOS – as seems to be the case with some Python modules – and the user needs to step in with some manual workarounds.

Documentation would likely help with that other case. If pydbus and dasbus had in their longDescription strings some warning that they require certain runtime information, the symptoms of what it looks like when it’s missing, such as error messages, and steps to take to fix the problem (e.g., set an environment variable in configuration.nix for a quick-and-dirty solution, use some kind of wrapper like wrapGAppsHook* if packaging an app that uses pydbus, maybe do something with nix-shell (?) if using those Python modules, etc.) Things would have been so much easier for me if Googling the error message would point me to the description of packages with that kind of long description!

It wouldn’t even be that hard for the packager to test for likely use cases. For a Python module in particular, seeing what happens at a REPL when a user uses the module according to its documentation would be an obvious go-to. For a GNOME extension, well, try it and see if it works.

If the NixOS developers are going to insist on avoiding global state, they can at least also insist on documenting workarounds so that users don’t have to waste time with guesswork.

1 Like
12

Sounds reasonable to me. I’ll be happy to see what you can dig out.

13

Writing a derivation for a small script really doesn’t have to be hard. There are lots of shortcuts available in nixpkgs for simple things. For example, with python, you can do something like this:

pkgs.writers.writePython3 "myscript" {
  libraries = with pkgs.python3Packages; [ numpy ];
} ./myscript.py

(Use writePython3Bin if you want the output to be formatted like a package with a bin directory, rather than a bare script in the nix store.)

You can also use a nix-shell shebang to make standalone scripts grab their environment through nix at the time they’re run. You can even combine the two approaches, so that myscript.py has a nix-shell shebang for rapid iteration during development of the script itself, which is then overwritten by pkgs.writers.writePython3 when used normally as part of your configuration.

Many ecosystems have similar things available.

1 Like
14

Thanks. From what I can tell, that particular script would still fail in my use case, since the problem is that dasbus needs a path to Glib’s typelib files. That doesn’t mean a script couldn’t be written for my use case, but it probably wouldn’t be as simple as yours.

[ETA: Googling for writePython3Bin didn’t help much. Looks like there isn’t any official documentation for it, though there is unofficial stuff at Nix-writers - NixOS Wiki. Also, looks like you’re using a feature I haven’t seen in unofficial documentation: using a path to a script instead of a string with the contents of a script. Not sure if wrapGAppsHook3 would be allowed in the argument of libraries.]

BTW, here are the scripts I’d be running, and the context in which I’d run them: workrave/contrib/waybar-yambar-poss-other-applets at main · rcaelers/workrave · GitHub

And again, the big problem is deducing what missing state is causing breakage, and the time sink that entails. Near as I can tell, the only fix for that which is consistent with the “Nix way” is for packagers to test their packages, find how they fail, and then document the failure modes and workarounds.

15

Yep. I’d consider that more of an example, your case is still pretty simple - it’s an alternate way to implement the docs @waffle8946 linked earlier:

let
  gi_typelib_path = lib.makeSearchPath "lib/girepository-1.0" [
    pkgs.glib
    # Any other packages with gobjects you want to introspect
  ];
in
  pkgs.writers.writePython3 "myscript" {
    libraries = with pkgs.python3Packages; [ numpy ];
    # If you read the docs I link below, you can see that `makeWrapper`
    # is exposed to all script writers like this.
    #
    # Unfortunately, `wrapGapps` *isn't* exposed, so we need to do
    # this "manually". This may or may not become awkward depending
    # on how involved your script actually is, at which point you should just
    # make a custom derivation - which isn't much more complex tbh, this
    # maybe saves 3 lines.
    makeWrapperArgs = [
      ''--prefix GI_TYPELIB_PATH : "${gi_typelib_path}"''
    ];
  } ./myscript.py

There is, unfortunately finding docs for many utility libraries requires actually looking inside the nixpkgs repo.

Google is universally unhelpful, especially these days, it’s become very enshittified. I really recommend having a clone of nixpkgs locally and getting good with ripgrep and/or a good editor with nice search functionality.

Should there be a good index into all nixpkgs functions with excellent SEO so that Google just magically works? Absolutely, unfortunately that doesn’t currently exist.

This is specifically a (pretty well-documented, once you find it) feature of the generic makeScriptWriter, which writePython3 is a specialization of.

You’re also welcome to add these docs upstream as you run into issues like this. It’s difficult for nixpkgs contributors to know what issues users will run into in practice, and what is not obvious to them. Most python packages are created semi-automatically anyway, so there’s nobody who would really have the level of ownership over pydbus specifically to realize this is needed; it’s just a very specific configuration. Most python libraries are just there because they are dependencies of other packages, and not really designed to be used for programming directly.

Plus, as your familiarity with glib, python, and nix grows, most of these types of errors become very obvious, making it easy to overlook the need for docs. And honestly, upstream has its hands full just keeping the lights on; cut them some slack.

The utility wrappers/scripts/etc. of nixpkgs absolutely do need more than just the in-code docs, and the docs that do exist probably do need some work too, it’s often not transparent what args do without reading the code. AIUI there’s some work in that direction (which is why the in-code docs are at least decent nowadays), but large parts are still underdocumented, and the reference docs remain very poorly structured.

Acknowledging that the current state isn’t perfect doesn’t really help you though, which is why pragmatically the only answer is to ask around here and/or gain experience yourself to the point that this stuff is trivial to you. And to read nixpkgs source code; once you become comfortable just doing that instead of assuming Google always has an answer the ecosystem really opens up a lot.

1 Like
16

Is there any automated testing of the packages to make sure that they work at runtime? If you had a test that even just ran the line “import pydbus”, the problem would be caught pretty quickly, as that line alone leaves a conspicuous error message: ImportError: cannot import name Gio, introspection typelib not found.

For dasbus, it would be slightly more complicated because “import dasbus” would work, but “from dasbus.connection import SessionMessageBus” and “from dasbus.connection import SystemMessageBus” would fail – and those latter lines of code are in the dasbus examples. Again, the failure is conspicuous, with the error message ValueError: Namespace GLib not available.

These failures are not subtle, and even cursory testing could find them. If the packagers found those kinds of errors first, then traced down their sources and documented them, it would save a lot of grief.

1 Like
17

pydbus has:

  pythonImportsCheck = [
    "pydbus"
    "pydbus.generic"
  ];

which ought to be doing exactly that.

18

“ought” is the operative word here.

19

If you can demonstrate that pythonImportsCheck isn’t functioning correctly, that would be a useful bug report.

20

Looks like what’s happening is that the imports are done within the environment set by the derivation.

One thing I notice is that if I run a Python script that depends on pydbus within a shell started by nixshell -p python3Packages.pydbus, then the script actually works. Which is what @tejing was trying to tell me when he was talking about nix-shell shebangs, but I didn’t quite get it then.

I think I’m finally getting at least a good chunk of the answer to this question:

It looks to me like the “Nix way” for avoiding dependency problems in Python (and probably Perl and R as well) can be described as “Never run a naked interpreter”. Replace instances of #!/usr/bin/env <my_interpreter> with #!/usr/bin/env nix-shell followed by #!nix-shell -i <my_interpreter> -p <python module 1> <python module 2> .... Alternatively, use Python distributions like Anaconda, conda-forge, etc., that provide their own ways of handling environments and dependencies (though I’d personally tend to reserve those for more scientific uses). That’s at least making sense to me.

That doesn’t solve my problems with the Workrave GNOME extension, but that’s at least as much of a packaging issue as anything.

1 Like