`programs.sqlite`, `nix-index`, `nix-locate`, `comma` oh my

1

Starting in a new topic, since it was getting off-topic

From what I can tell:

command-not-found is a nixos feature which allows you to type a command name, like lscpu, and if you don’t have that command available in your current system profile/generation, it will tell you what available NixOS packages contain an executable of the same name.

It relies on programs.sqlite, which is generated as part of the channel release process by scripts available at GitHub - NixOS/nixos-channel-scripts: Tools to create channels from hydra jobsets. Because generating this database requires building essentially all programs, the database is not available from a github checkout since it inherently relies on hydra builds. (EDIT: blurred because not quite sure it’s accurate: This is part of why the channel-based expressions available on https://releases.nixos.org/ take a few days to be available after the channel tag gets updated on github (since the tag updating is gated on a smaller number of tests passing, and so completes earlier).) This is all better described at Channel branches - NixOS Wiki.

nix-index is a third-party tool to provide the same functionality in another way. It is best described by it’s git page:

nix-index is a tool to quickly locate the package providing a certain file in nixpkgs. It indexes built derivations found in binary caches.

nix-index provides nix-locate, which is a command that also lets you see which packages contain a specific executable. nix-index-database provides pre-generated databases and a wrapper around the whole thing that makes it easier to use declaratively. nix-locate is more explicit in it’s intent than just typing a command at the CLI, and therefore more amenable to scripting.

I don’t really know why you’d prefer to use nix-index over the builtin https://release.nixos.org option, but it does exist. I suppose I also don’t really know why you’d prefer to use the https://release.nixos.org option either. They seem functionally very equivalent, but I am not an expert. EDIT: it seems that although providing similar functionality, it’s worth noting that the actual database file you get is not the same, per the replies below. As tejing notes, nix-index also provides dynamic library information, so this is one reason to use nix-index.

comma is a tool which uses programs.sqlite EDIT: nix-index to spawn ephemeral shells which will contain an executable, so it’s a useful convenience and another reason to use nix-index.

Note that if you are not using nix-channel the tool, you will need to set the command-not-found.dbPath appropriately for programs.sqlite.

Are there other cool functionality that people are aware of that rely on programs.sqlite or nix-index?

4 Likes
2

Note that there are two different database types at play here.

One type is used by the built-in command-not-found script in nixos. This is distributed through the channel infrastructure, and I don’t know any other way to get it. It has limited information, however, only having an association to commands in /bin inside the packages.

For this database, you either need to use channels, or get creative, and I’ve packaged up a fairly universal solution for non-channel situations here, which downloads the actual programs.sqlite from the channel infrastructure at runtime. As long as you have a way to specify a channel and a git rev that’s listed for that channel, it can download it for you, and automatically update when the declarative rev changes.

The other type is that generated by nix-index, which you can avoid having to do locally through the automatically-updated nix-index-database. This is the database used by nix-locate and comma, and it contains a more general listing of all the files in the output store paths, so it can be used for more than just commands. In particular, it can be helpful for finding out which package provides a particular dynamic library.

4 Likes
3

What you get on the channel is built by ${nix-index}/bin/nix-channel-index (done by this script).

3 Likes
4

I got creative a while ago: Flake-programs-sqlite: fixing command-not-found for pure flake systems
As I don’t get many complaints it either has no users (aside from myself) or it just works…

1 Like
5

Surely a better fix would be to change the upstream default to the tarball location if the flake input uses the tarball, and to recommend people use the tarball. Then you get the precise database for your package set, rather than a close approximation.

6

I think you’ve confused things here, the nixos branches take a few days longer than the nixpkgs branches. The tarballs should be released pretty much at the exact same time as the nixos commits. The wiki page you link to says as much, as well.

1 Like
7

It’s entirely possible that I’m confused! There was a situation where I seem to remember seeing the github tag for nixos-unstable progress but I wasn’t able to download the release for a few days, and updating the programs.sqlite index is what made sense to me when I was investigating why. It’s also possible that it was just some sort of hydra hiccup or that I misconfigured something at that time or even just that I’m misremembering. It was (even) earl(ier) days haha.

8

Maybe? Is the non-tarball only a “close approximation” if you generate the index locally via nix-index? Like I said, not an expert.

I’m an npins user and I do use the tarball and use pkgs.outPath as part of the link the correct programs.sqlite per generation. As I said in my original post

9

This is specifically in response to @wamserma 's workaround, which:

flake-programs-sqlite makes use of a mapping from git commit hashes to channel names. This mapping is updated by a GitHub action by scraping {channels,releases}.nixos.org (currently) about every ten minutes.

AIUI this means you can end up with a commit that doesn’t precisely map to a command-not-found db, but that might not be true, in fairness.

1 Like
10

Aside from a few occasions where a channel was released with a broken DB, only commits with a DB are recorded by flake-programs-sqlite. If you happen to use a commit that is does not match a bump of a channel, a sensible fallback is used.

11

Approximation should be good in practice. I believe it’s rare to run into relevant changes (packages or bin/ files getting renamed or moved).

12

Sure! But… We have a perfectly accurate database right there, why work around a limitation we can just eliminate?

1 Like
13

I’m confused. I thought it’s about not having this database ready for every commit, so you find a close commit for which we do have it.

14

Well it’s for finding any file, which is why it’s useful (and basically the only option) for cases where you want to find a library, or an icon, or whatever else.

command-not-found is also slow, which is why even for finding executables, I use nix-locate every time.

It’s a workaround for this alleged problem (channel advances but they thought that the tarball didn’t, so they pick an older tarball instead). If they’re released basically concurrently, though, this should be a non-issue, and the workaround would be unnecessary.

If you’re using an arbitrary commit, then sure, this workaround makes sense; but how many people do so (intentionally)?

1 Like
15

That’s weird. Generating this DB happens during the channel update process. Errors would prevent the channel from advancing. If you (anyone) have some concrete examples on that, we in the Infra team can look at it (logs etc.)

1 Like
16

Does a flake input specified like that actually allow access to old expressions when you don’t update your lockfile? I would assume it doesn’t. Which makes it a non-starter for me.

17

Why wouldn’t it allow that? Lockable tarballs are a thing.

$ cat flake.nix
{
  inputs = {
    nixpkgs.url = "https://channels.nixos.org/nixos-unstable/nixexprs.tar.xz";
  };

  outputs = _: {};
}

$ nix flake metadata
Resolved URL:  path:/tmp/tmp.p8USHBfGyR
Locked URL:    path:/tmp/tmp.p8USHBfGyR?lastModified=1760985208&narHash=sha256-fOMA%2BRP%2BKaxc7708i4NWyZmZIcuuobf44rSzmCjl6VM%3D
Path:          /nix/store/4d7a2m07xch1kd8d1m5x3798ha3d1b74-source
Last modified: 2025-10-20 17:33:28
Inputs:
└───nixpkgs: https://releases.nixos.org/nixos/unstable/nixos-25.11pre880095.5e2a59a5b1a8/nixexprs.tar.xz?narHash=sha256-u0JUo46QSoXnjLaezAM75wRNuxVMVbm5OxHH122TeTY%3D&rev=5e2a59a5b1a82f89f2c7e598302a9cacebb72a67
    Last modified: 2025-10-19 20:54:49
1 Like
18

Oh. I had no idea that existed. Cool.

20

Since fastly: implement "Lockable HTTP Tarball Protocol" (Flakes) for channels.nixos.org by emilylange · Pull Request #562 · NixOS/infra · GitHub, you should be able to use

  inputs = {
    nixpkgs.url = "https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz";
  };

in flake.nix. Which contains programs.sqlite directly

21

I can start a new thread if this is too far afield, but on the topic of programs.sqlite and searching nixpkgs for outputs, have we thought about adding package metadata that declares outputs in advance? If .meta contained a list of expected files, then it would only require evaluating nixpkgs, rather than building it, to find derivations that produce a given file. It would also automatically play nice with overlays if the overlaid packages use the same .meta convention. We could add a build hook to verify that the list of output files matches what’s actually built, and that would enforce keeping it up to date.