Nix-search: A faster, better nix search

Announcements
1

Since Nix 2.4’s release, I’ve always been fed up at how nix search has become so much slower than Nix 2.3’s search, and everyone kept telling me to use search.nixos.org! So I made a really fast nix search alternative called nix-search:

image
image1380×928 165 KB

asciicast

The goal of nix-search is to be as similar to Nix 2.3’s nix search as possible while also not taking a very long time to index and search. For example, on my computer, indexing takes ~20s while searching takes way less than 1s:

―❤―▶ time nix-search --index

real	0m21.760s
user	2m36.436s
sys 	0m30.729s

―❤―▶ time nix-search firefox > /dev/null

real	0m0.033s
user	0m0.028s
sys 	0m0.006s

nix-search supports both Nix flakes and channels, but it uses the <nixpkgs> channel by default. You may override this by doing --channel '<nixpkgs>' or --flake nixpkgs when indexing.

nix-search also adheres to $PAGER, so you can easily use your favorite pager. For example, to use nix-search with most:

nix-shell -p most
PAGER=most nix search firefox

Internally, nix-search is powered by an actual search database called Bluge, so performance is really good while the searching itself is very flexible. For example, you may run nix-search with --exact=false which turns off the exact match filtering:

―❤―▶ nix-search --exact=true python | head -n2  # default
- nixpkgs.pythonFull (2.7.18.8)
  A high-level dynamically-typed programming language

―❤―▶ nix-search --exact=false python | head -n2
- nixpkgs.pyp (1.2.0)
  Easily run Python at the shell

Installation

Installation (non-Flakes)

nix-search comes with a default.nix that has itself packaged. To install it, simply fetch it from GitHub and import it:

let
  nix-search = import (pkgs.fetchFromGitHub {
    owner = "diamondburned";
    repo = "nix-search";
    rev = "<REV>";
    sha256 = "<SHA256>";
  });
in

{
  environment.systemPackages = [ nix-search ];
}
Installation (Flakes)

nix-search comes with a flake.nix that contains itself in nix-search.packages.${system}.default:

git+file:///home/diamond/Scripts/nix-search?ref=refs/heads/main&rev=98c54882df337dc15387ac725543784a31f64935
└───packages
    ├───aarch64-darwin
    │   └───default omitted (use '--all-systems' to show)
    ├───aarch64-linux
    │   └───default omitted (use '--all-systems' to show)
    ├───x86_64-darwin
    │   └───default omitted (use '--all-systems' to show)
    └───x86_64-linux
        └───default: package 'nix-search-98c54882df337dc15387ac725543784a31f64935'
Installation (using Go)

Make sure that your $GOBIN is in your $PATH, then run:

go install libdb.so/nix-search/cmd/nix-search@latest

GitHub Repository: diamondburned/nix-search

34 Likes
2

Thanks, I was just looking for something like this the other day!

It wasn’t clear to me from a quick skim of your code – do you know about the packages.json.br output of the tarball job? That lets you delegate all the work of actually evaluating the package set to Hydra, so it should hopefully be considerably faster to index if your current approach involves calling into Nix proper.

5 Likes
3

I actually had no idea… Currently, I’m invoking nix for every attrset and using goroutines to parallelize indexing them.

This Hydra output seems really cool! Can I assume that there will be a Hydra output for every Nixpkgs commit, or is there a specific rule for that?

4

My impression of you is still of the dissent developer, but didn’t expect you also have done many other interesting tools! Appreciate it very much!

2 Likes
5

Every Nixpkgs and NixOS channel update is gated on the tarball job, which is part of the channel release process. So as long as someone is on a revision that has been part of a channel, there should be a corresponding packages.json.br cached. However, you don’t need to think about Hydra at all; you should just be able to try to build it from release.nix, which will be substituted as long as all the parameters match up correctly or built as necessary, like any other package. Although that would end up also downloading the actual tarballs as well… I don’t know if there’s a particularly nice interface to get a hydra-build-products file, but someone who is more familiar with Hydra than me might.

3 Likes
6

This is actually very cool! I’ve been wanting to look for something that could give me seach.nixos.org like output locally, espcially inside emacs, is it possible to add json output this? Then I could invoke it from Emacs and format it nicely into a buffer :smiley: . But in any case, looks great, I’ll try it out and run it with async-shell-command for now

1 Like
7

There’s also this, by the way, in case anyone is like yesterday‐me and hadn’t heard of it before: GitHub - peterldowns/nix-search-cli: CLI for searching packages on search.nixos.org. That’s just querying over the network, though.

8

v0.3.0 has --json now!

9

Not sure if that is what you meant, but you can download packages.json.br from https://releases.nixos.org/ with corresponding channel and commit, thought it’s quite big (200-300mb).
Current latest nixos-unstable:
nixos-unstable release nixos-24.11pre647193.9f4128e00b0a
direct link to packages.json.br
(do not click in browser as is, at least firefox tries to open it instead of downloading by default and lags):
https://releases.nixos.org/nixos/unstable/nixos-24.11pre647193.9f4128e00b0a/packages.json.br

10

After trying just about every alternative I know of, even rippkgs and nh, nothing seems to come close to this one in thoughtful design and performance. Thank you for making and sharing this!

1 Like
11

Is this nix-index and nix-locate ?

Seems closer to those two tools than “nix search”

12

Not quite. nix-index is meant to search for paths in packages, while nix search searches the package list itself. The only similarity to nix-index is the fact that it indexes, but that’s normal to do.

2 Likes
13

It’s actually under 6 MiB to download, smaller than the Nixpkgs checkout itself; it just inflates massively in size when decompressed. You’d want to stream the decompression directly into the indexer.

14

Hi, I tried to install this as non-flake via home manager, but it doesn’t seem to add any binaries to path. My code is:

{ pkgs, ... }:

let
  nix-search = import (pkgs.fetchFromGitHub {
    owner = "diamondburned";
    repo = "nix-search";
    rev = "v0.3.1";
    sha256 = "sha256-2N1xy9BK9o7j/vk42Pg/nvDjDt7nZlannCLCcS8Puuk=";
  });
in
{
  home.packages = [ nix-search ];
}

Doesn’t add any program called nix-search.

15

Does this also improve the memory usage? nix search has a maximum resident set size (measured with time -v) of 4GB…

16

Not a lot here!

Command being timed: "nix-search firefox"
User time (seconds): 0.02
System time (seconds): 0.00
Percent of CPU this job got: 2%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.12
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 14080
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 1850
Voluntary context switches: 224
Involuntary context switches: 16
Swaps: 0
File system inputs: 256
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
17

Thanks! What about nix-search --index?

18

That gave me Maximum resident set size (kbytes): 2530016, though I’m wondering if this includes subprocesses as well, as nix-search --index works by shelling out to nix for most of its work.

19

Half the size, nice :slight_smile:

It does include subprocesses, as it should imo.

I think search would benefit from a custom memory management strategy. What probably happens is that nix keeps all evaluated metas in memory in case it will need them again even though in this case the chance of reusing them is negligible. Evaluating metas one by one instead of all at once could reduce memory usage with only a small performance impact.

1 Like
20

Right, and I was surprised to see the index gone as well, as it makes complete sense for something like search!