Help understanding package paths in /nix/store

Help
1

I’m trying to learn NixOS with Flakes by setting up a little game emulator on a raspberry pi using emulationstation.
I’m trying to provide emulationstation with the path to different cores to be used, for example snes9x_libretro.so
So if I run find, I see the results

sudo find /nix/store -name '*snes9x_libretro.so'
/nix/store/v4w2gqxxa97x413006w2fls0d4wjj0vp-retroarch-with-cores-1.16.0.3/lib/retroarch/cores/snes9x_libretro.so
/nix/store/9cg9a1aks0blibm7ffm50s2vdx9fdkqd-retroarch-with-cores-1.16.0.3/lib/retroarch/cores/snes9x_libretro.so

But if I try to insert that path into a configuration file like

retroArchCorePath = "${pkgs.retroarchFull}";

retroArchCorePath ends up evaluating to

/nix/store/1s3d0zr4hhykp154wy95vsbynxrbw0v6-retroarch-with-cores-1.16.0.3/

which exists but doesn’t contain a /lib/retroarch/cores/snes9x_libretro.so that I’m looking for.

How/why would there be multiple retroarch-with-cores-1.16.0.3 and why would some contain a lib folder and others not?
How can I reliably refer to the package that contains the cores?

Note, if helpful: I’m running nix-collect-garbage -d regularly also to be sure find isn’t picking up old derivations or something. So I’m pretty sure there are multiple retroarch-with-cores-1.16.0.3 from a single nixos-rebuild switch ...

Thanks

2

I am curious as well. I wonder if it could be related to symlinkJoin.

defined here: https://github.com/NixOS/nixpkgs/blob/d374eafed64d5fc891626ebd4b8804b41f6c4891/pkgs/build-support/trivial-builders/default.nix#L533

used here: https://github.com/NixOS/nixpkgs/blob/d374eafed64d5fc891626ebd4b8804b41f6c4891/pkgs/applications/emulators/retroarch/wrapper.nix#L29

Perhaps the same name is being used for one of the things being joined, and also the result of their union?

3

Thanks. That does sound like a solid theory. Unfortunately, I’m not familiar enough at this point with Nix to really debug that retroarch code.

I’m not really sure how to debug where all the different retroarch-with-cores folders come from within my flake or how to refer to one of them specifically.
If ${pkgs.retroarch} points to the wrong folder, is there a way to list the other paths available such that ${pkgs.XYZ} points to the correct one?

4

That’s a familiar feeling :smile: I’m getting there slowly. One other thing I notice is that two packages appear in the search: NixOS Search

There’s retroarchFull and retroarch, both indicate the name: retroarch-with-cores and both link to the same source file, but they have separate descriptions.

Screen Shot 2023-12-10 at 11.34.30 AM
Screen Shot 2023-12-10 at 11.34.30 AM1702×716 78.8 KB

I don’t know for sure, but it’s starting to smell like a bug. Maybe one of the maintainers can shed some light:

Any ideas @edwtjo @k0kada @matthewbauer ?

1 Like
5

When I looked in pkgs/applications/emulators/retroarch/cores.nix I noticed that snes9x.meta.license = lib.licenses.unfreeRedistributable;, and IIRC those aren’t yet built and available via cache.

(https://github.com/NixOS/nixpkgs/issues/83884 suggests this is still the state of things.)

I’m not sure how that would manifest in this case where a bunch of free and ~unfree derivations are getting composed together, but I imagine this has something to do with configuring allowUnfree or allowUnfreePredicate?

6

Thanks for the context.
I was thankfully able to get past my blocker. It was (as expected) a total newbie mistake. Once I added my current state to git, the paths resolved correctly and I was able to get things working. I’m on to new errors to solve :slight_smile:

I still do have 3 /nix/store/<hash>-retroarch-with-cores-1.16.0.3 folders that are not identical but at least ${pkgs.retroarchFull} resolves to the one I need.

7

I’m glad you got your problem resolved, @jaylward, but I was wondering if you might explain a bit more what you did to fix it. I didn’t understand what you meant by your quote above.

8

I forgot that, when using flakes, you need to add all your files to git before rebuilding. Mentioned in the Git Warning here.
In my case, the "${pkgs.retroarchFull}" was not resolving correctly because (I think) the expected package was not being copied over into the /nix/store and whatever other retroarch folders that l found using find were unrelated.
I’m still pretty confused by the whole thing but hopefully that helps. Basically, when using flakes, git add . before sudo nixos-rebuild switch --flake /etc/nixos

1 Like
9

This isn’t that uncommon; there’s no guarantee that a package’s name is unique in Nixpkgs, and the source link that shows up in Nixpkgs search is only smart enough to point to wherever the meta.description attribute is defined, which is often (as it is in this case) a wrapper script file used in multiple packages.

If you have two package attributes (retroarchFull and retroarch, here) and you want to know how they’re different, the thing to do¹ is to look in pkgs/top-level/all-packages.nix and compare those definitions. In this case, we have:

  retroarchFull = retroarch.override {
    cores = builtins.filter
      # Remove cores not supported on platform
      (c: c ? libretroCore && (lib.meta.availableOn stdenv.hostPlatform c))
      (builtins.attrValues libretro);
  };

  retroarch = wrapRetroArch {
    retroarch = retroarchBare;
    settings = {
      assets_directory = "${retroarch-assets}/share/retroarch/assets";
      joypad_autoconfig_dir = "${retroarch-joypad-autoconfig}/share/libretro/autoconfig";
      libretro_info_path = "${libretro-core-info}/share/retroarch/cores";
    };
  };

From the use of override, we see that retroarchFull is exactly retroarch, but with all supported cores added from libretro (retroarch itself defaults to an empty list of cores, a fact you can learn from reading the wrapper script linked from the Nixpkgs search).

So it seems not to be a bug that this package appears twice. They are indeed two different packages.

¹ If you’re reading this in the future, the packages you’re looking for may not be in that file anymore. There’s a new organization scheme slowly being applied to Nixpkgs, in which package files are placed under pkgs/by-name. The thing to do then would be to compare the two package files in the directories corresponding to the attribute names.

2 Likes
10

Thanks for the explanation @rhendric, that makes sense.

If I’ve understood, then what’s weird here is that both are called retroarch-with-cores when in fact only one has the cores.

1 Like
11

You’re not wrong! Package names are generally a little weird, though. They used to be a primary way you’d reference a package to install it using nix-env, but nowadays imperative use of nix-env is strongly discouraged and there’s much more emphasis on using attribute paths to identify a package. I think package names are sort of left hanging around because they’re somewhat useful for identifying what a store path is, but they don’t get much care applied to them because people aren’t interacting with them directly.

I actually think the Nixpkgs search app shouldn’t display them at all, but that might be a spicy take.

1 Like
12

This has been super helpful as a newbie but I’m still super lost on how to reason through how to navigate packages.
For example, I have something like

home.packages = with pkgs; [
    emulation-de
    retroarch
];
...
nixpkgs.config = {
  allowUnfree = true;
    retroarch = {
    enableSnes9x = true;
  };
};

and I need to manually refer to where that snes9x is installed, I can do a find /nix/store | grep '*libretro_snes9x.so' and I see multiple results inside the nix store. How do I programmatically resolve those paths?
I asked ChatGPT because it’s been surprisingly useful in learning NixOS and it says
retroArchCorePath = "${pkgs.retroarch}/lib/retroarch/cores/"; but that ends up resolving to a different retroarch folder in the /nix/store.

Since ${pkgs.retroarch} is a legitimate expression, is there any way to see what other names are available, ie ${pkgs.XYZ} or ${pkgs.retroarch.XYZ}`. I really don’t understand how to infer these things yet, like listing what’s available within the language.

I feel like if I could get over this hump, I could get a lot more accomplished within the whole ecosystem. Like there’s an API document that I’m missing.

13

I don’t see it documented anywhere, but I checked the code and it looks like these are exposed as a set named libretro (so you should be able to reference pkgs.libretro.snes9x, and a NixOS package search for libretro. should show everything available):

This set of cores is what gets passed in to retroarchFull:

I imagine you picked up this config pattern from someone’s config or a post, but it looks like this approach was removed back in retroArchCores: remove, retroarchFull: init, retroarch: use fixed paths on "libretro_info_path" by thiagokokada · Pull Request #146714 · NixOS/nixpkgs · GitHub

If you just need snes9x, it looks like you could use something like:

home.packages = with pkgs; [
    emulation-de
    (retroarch.override { cores = with libretro; [ snes9x ]; })
];
...
nixpkgs.config = {
  allowUnfree = true;
};
1 Like
14

Better way:

nix-store -q --requisites /var/run/current-system | \
  tr '\n' '\0' | \
  find -files0-from - -name 'libretro_snes9x.so'

That’ll get you just the files that are reachable from the system configuration you’re currently running, so you don’t get duplicates from older generations and you don’t have to be constantly garbage collecting.

The online Nixpkgs search is useful when I have no idea where to start.

I don’t know how people who use flakes do it, but for those of us who use channels, the tab completion in nix repl is great for this:

$ nix repl                                                                                                   
Welcome to Nix 2.13.6. Type :? for help.

nix-repl> pkgs = import <nixpkgs> {} 

nix-repl> pkgs.libretro.<TAB>
pkgs.libretro.atari800                   pkgs.libretro.bsnes-mercury-balanced     pkgs.libretro.hatari                     pkgs.libretro.o2em                       pkgs.libretro.snes9x2002
pkgs.libretro.beetle-gba                 pkgs.libretro.bsnes-mercury-performance  pkgs.libretro.mame                       pkgs.libretro.opera                      pkgs.libretro.snes9x2005
pkgs.libretro.beetle-lynx                pkgs.libretro.citra                      pkgs.libretro.mame2000                   pkgs.libretro.override                   pkgs.libretro.snes9x2005-plus
pkgs.libretro.beetle-ngp                 pkgs.libretro.desmume                    pkgs.libretro.mame2003                   pkgs.libretro.overrideDerivation         pkgs.libretro.snes9x2010
pkgs.libretro.beetle-pce-fast            pkgs.libretro.desmume2015                pkgs.libretro.mame2003-plus              pkgs.libretro.parallel-n64               pkgs.libretro.stella
pkgs.libretro.beetle-pcfx                pkgs.libretro.dolphin                    pkgs.libretro.mame2010                   pkgs.libretro.pcsx-rearmed               pkgs.libretro.stella2014
pkgs.libretro.beetle-psx                 pkgs.libretro.dosbox                     pkgs.libretro.mame2015                   pkgs.libretro.pcsx2                      pkgs.libretro.swanstation
pkgs.libretro.beetle-psx-hw              pkgs.libretro.eightyone                  pkgs.libretro.mame2016                   pkgs.libretro.picodrive                  pkgs.libretro.tgbdual
pkgs.libretro.beetle-saturn              pkgs.libretro.fbalpha2012                pkgs.libretro.melonds                    pkgs.libretro.play                       pkgs.libretro.thepowdertoy
pkgs.libretro.beetle-snes                pkgs.libretro.fbneo                      pkgs.libretro.mesen                      pkgs.libretro.ppsspp                     pkgs.libretro.tic80
pkgs.libretro.beetle-supafaust           pkgs.libretro.fceumm                     pkgs.libretro.mesen-s                    pkgs.libretro.prboom                     pkgs.libretro.vba-m
pkgs.libretro.beetle-supergrafx          pkgs.libretro.flycast                    pkgs.libretro.meteor                     pkgs.libretro.prosystem                  pkgs.libretro.vba-next
pkgs.libretro.beetle-vb                  pkgs.libretro.fmsx                       pkgs.libretro.mgba                       pkgs.libretro.puae                       pkgs.libretro.vecx
pkgs.libretro.beetle-wswan               pkgs.libretro.freeintv                   pkgs.libretro.mkLibretroCore             pkgs.libretro.quicknes                   pkgs.libretro.virtualjaguar
pkgs.libretro.blastem                    pkgs.libretro.gambatte                   pkgs.libretro.mupen64plus                pkgs.libretro.recurseForDerivations      pkgs.libretro.yabause
pkgs.libretro.bluemsx                    pkgs.libretro.genesis-plus-gx            pkgs.libretro.neocd                      pkgs.libretro.sameboy
pkgs.libretro.bsnes                      pkgs.libretro.gpsp                       pkgs.libretro.nestopia                   pkgs.libretro.scummvm
pkgs.libretro.bsnes-hd                   pkgs.libretro.gw                         pkgs.libretro.np2kai                     pkgs.libretro.smsplus-gx
pkgs.libretro.bsnes-mercury              pkgs.libretro.handy                      pkgs.libretro.nxengine                   pkgs.libretro.snes9x
3 Likes
15

Wow. Those are exactly the kinds of things I have been looking for. Thank you.

2 Likes
16

This works great regardless of whether or not one uses channels/flakes, I think… What a brilliant idea. Thank you!