Handy scripts for fuzzy searching nixpkgs and nixos-options

#1

Hey y’all, I cooked up a few little scripts to help make searching for package names and options a bit easier; I thought I would share the results in case it was useful to others, or to get feedback.

This script can be adapted for a ton of other uses, but here is how I have it wired together:
an XMonad hotkey launches a utrvt terminal, running fzf, this provides fuzzy searching on packages or options, using an fzf preview window to see additional information (current settings or package descriptions).
Here are the results:


First we need to generate 2 text files full of all the system’s packages and options:

  1. Packages are real easy, simply do nix-env -qa -P | awk '{print $1}' > allpackages.txt
  2. Options are a bit harder, I eventually made a short haskell program to wrap around the “nixos-option” command to dump every possible option name, I am sure there is a much better solution to this. But here is my script:
{-# LANGUAGE OverloadedStrings #-}

module Main where

import qualified Data.Text.Lazy as T
import Data.Text.Lazy (Text)
import qualified Data.Text.Lazy.IO as TIO
import Data.Tree
import System.IO
import System.Process
import Control.Monad
import Data.Monoid

-- Builds option list from local nixpkgs (including new modules)

nixQuery :: Text -> IO Text
nixQuery q = withFile "/dev/null" WriteMode (\null -> do
  (_, Just hout, _, _) <- createProcess (proc "nixos-option" [T.unpack q])
    { std_out = CreatePipe, std_err = UseHandle null }
  TIO.hGetContents hout)

nixBranch :: Text -> IO (Text, [Text])
nixBranch q = do
  ls <- nixQuery q
  if T.isPrefixOf "Value:" ls then return (q, []) else
    let ls'  = T.lines ls
        lsd  = if T.null q then ls' else map ((q<>) . T.cons '.') ls'
    in return (q, lsd)

mkTree :: Text -> IO (Tree Text)
mkTree = unfoldTreeM nixBranch

leaves :: Tree a -> [a]
leaves (Node x xs) | null xs   = [x]
                   | otherwise = concatMap leaves xs

options :: Text -> IO [Text]
options q = leaves <$> mkTree q

main = TIO.writeFile "alloptions.txt" . T.unlines =<< options ""

even looking at that script again I see a few things that would make it significantly faster. Piping directly to the file instead of using “writeFile” would be a big improvement. But in any case this is functional. Compiling with “-threaded” helps a bit, but full disclosure: this is slow as hell and needs to be rebuilt from the ground up. When I set out I had imagined I was going to to hold onto this tree structure, but in actuallity it shouldn’t construct a tree at all. Instead it should just crawl through the options and immediately write them to the text file when it reaches a dead end. IN ANY CASE IT WORKS haha

Edit: I have made a solution with Nix that is a million times faster. It can be found here for anybody who wants to use it.

to run fzf I use:
packages

cat allpackages.txt | fzf \
    --reverse \
    --prompt="NixOS Packages> " \
    --preview="echo -e \"{1}\n\"; nix-env -qa --description -A {1]" \
    --preview-window=wrap

and for options

cat alloptions.txt | fzf \
    --reverse \
    --prompt="NixOS Options> " \
    --preview="echo -e \"{1}\n\"; nixos-option {1}" \
    --preview-window=wrap

for folks who want to wrap all of is a urxvt launcher:
urxvt -e zsh -ic 'cat allfoo.txt | fzf ...'
Just remember to add extra escapes and what not where necessary.

Extensions:

  1. Obviously improving the haskell program
  2. Add fzf actions to open files that define packages and options.
  3. Colorize/otherwise improve fzf previews
7 Likes
#2

Cool. Your using Haskell to do this reminded me of how nixpkgs-update looks up the attr path given a package name. It also is pretty slow, I bet I could speed it up by caching the entire list of packages into a HashMap, like how you save them to allpackages.txt.

#3

Thank Ryan. Last night I caved and gutted the a bunch of the documentation code out of <nixpkgs/nixos/docs/manual> to accomplish this same task, which runs miiiiiles faster. It has the added advantage of dumping descriptions and other info as well so I don’t have to use nixos-options with fzf (which was a little sluggish).
I do prefer writing scripts in Haskell though, Nix is great for packages but it can be so frustrating when you want to borrow a function thats floating around some .nix file’s “let” section.
Nixly Solution

#4

Your screenshots look seductive. FZF is much more interesting than I expected. I imagine combined dictionary:

  • options (current value, default, defintion file, declaration file)
  • packages (description, defintion file, latest version, current version)
  • tests (defintion file)
  • filenames in nixpkgs source tree
  • … maybe some other runtime information (for current system)? …
2 Likes
#5

That sounds like a solid plan. Be sure to check out my other post where I have a vastly improved nix script for pulling the options list.
And yeah I totally agree, FZF is such an awesome tool. Over the years I always find new uses for it; I know it’s basically just fancy AG, but the interface is just so nice :slight_smile:

Franken-Script to generate NixOS Options docs : with custom modules
#6

I’m back with updates! :blush:

I have added dedicated scripts to dump package/option information, which automatically updates if it is older than the most recent NixOS build.
FZF launches a new terminal window, with the package/option definition in a read only Vim buffer when the user hits enter.

nix-search

#! /usr/bin/env bash

# See if options and packages are outdated
if [[ $1 == "-o" ]]; then
    TARGET="/etc/nixos/bin/options.txt"
    PR="NixOS Options> "
    PREVIEW="nixos-option"
    function seeDef () {
      nixos-option "${1}" | tail -n 2 | head -n 1 | cut -d "\"" -f 2
    }
else
    TARGET="/etc/nixos/bin/packages.txt"
    PR="Nixpkgs> "
    PREVIEW="nix-env -qa --description -A"
    function seeDef () {
      nix-env -qa --json -A "${1}" \
        | jq ".\"${1}\".meta.position" \
        | cut -d "\"" -f 2 | cut -d ":" -f 1
    }
fi

SELECTION=`cat ${TARGET}                                \
  | fzf --prompt="${PR}" --reverse                      \
        --preview-window=wrap:70%                     \
        --preview="echo -e \"{1}\n\"; ${PREVIEW} {1}"`

DEFPATH=`seeDef ${SELECTION}`
urxvtc -tr -sh 8 -e zsh -ic "vim -R -c 'nnoremap q :q!<CR>' ${DEFPATH}; zsh"

# Do updates to package/options lists
NIXAGE=`date -r /etc/nixos/common.nix +%s`
OPTSAGE=`date -r /etc/nixos/bin/options.txt +%s`
PACKSAGE=`date -r /etc/nixos/bin/packages.txt +%s`

if [[ ( $(( ($NIXAGE - $OPTSAGE) / 60)) -gt 5) ]];
then ./writeOpts & fi

if [[ ( $(( ($NIXAGE - $PACKSAGE) / 60)) -gt 5) ]];
then ./writePacks & fi

#vim: sh

writeOpts

#! /usr/bin/env bash

echo $(nix-instantiate - --eval --show-trace <<EOF

with import <nixpkgs/nixos> { };
let
    extraSources = [];
    lib = pkgs.lib;

    optionsListVisible =
        lib.filter (opt: opt.visible && !opt.internal)
        (lib.optionAttrSetToDocList options);

    # Replace functions by the string <function>
    substFunction = x:
        if builtins.isAttrs x then lib.mapAttrs (name: substFunction) x
        else if builtins.isList x then map substFunction x
        else if lib.isFunction x then "<function>"
        else if isPath x then toString x
        else x;

    isPath = x: (builtins.typeOf x) == "path";

    optionsListDesc = lib.flip map optionsListVisible (opt: opt // {
        description = let attempt = builtins.tryEval opt.description; in
            if attempt.success then attempt.value else "N/A";
        declarations = map stripAnyPrefixes opt.declarations;
        }
        // lib.optionalAttrs (opt ? example) {
            example = substFunction opt.example; }
        // lib.optionalAttrs (opt ? default) {
            default = substFunction opt.default; }
        // lib.optionalAttrs (opt ? type) {
            type = substFunction opt.type; }
        // lib.optionalAttrs
            (opt ? relatedPackages && opt.relatedPackages != []) {
                relatedPackages = genRelatedPackages opt.relatedPackages; }
        );

    genRelatedPackages = packages: let
        unpack = p: if lib.isString p then { name = p; }
                    else if lib.isList p then { path = p; }
                    else p;
        describe = args: let
            title = args.title or null;
            name = args.name or (lib.concatStringsSep "." args.path);
            path = args.path or [ args.name ];
            package = args.package or (lib.attrByPath path (throw
                "Invalid package attribute path '\${toString path}'") pkgs);
            in "<listitem>"
            + "<para><literal>\${lib.optionalString (title != null)
                "\${title} aka "}pkgs.\${name} (\${package.meta.name})</literal>"
            + lib.optionalString (!package.meta.available)
                " <emphasis>[UNAVAILABLE]</emphasis>"
            + ": \${package.meta.description or "???"}.</para>"
            + lib.optionalString (args ? comment)
                "\n<para>\${args.comment}</para>"
            + lib.optionalString (package.meta ? longDescription)
                "\n<programlisting>\${package.meta.longDescription}"
                + "</programlisting>"
            + "</listitem>";
        in "<itemizedlist>\${lib.concatStringsSep "\n" (map (p:
            describe (unpack p)) packages)}</itemizedlist>";

    optionLess = a: b:
        let
        ise = lib.hasPrefix "enable";
        isp = lib.hasPrefix "package";
        cmp = lib.splitByAndCompare ise lib.compare
            (lib.splitByAndCompare isp lib.compare lib.compare);
        in lib.compareLists cmp a.loc b.loc < 0;


    prefixesToStrip = map (p: "\${toString p}/") ([ ../../.. ] ++ extraSources);
    stripAnyPrefixes = lib.flip (lib.fold lib.removePrefix) prefixesToStrip;

###############################################################################

    # This is the REAL meat of what we were after.
    # Output this however you want.
    optionsList = lib.sort optionLess optionsListDesc;

    optionsJSON = builtins.unsafeDiscardStringContext (builtins.toJSON
        (builtins.listToAttrs (map (o: {
            name = o.name;
            value = removeAttrs o [
            # Select the fields you want to drop here:
                "name" "visible" "internal" "loc" "readOnly" ];
        }) optionsList)));

in optionsJSON
EOF
) | sed 's/\\\\/\\/g' | sed 's/\\"/"/g' | head -c -2 | tail -c +2 | \
    jq 'keys | .[]' | sed 's/^"\(.*\)"$/\1/g' > options.txt

# vim: ft=sh

writePacks

#! /usr/bin/env bash
nix-env -qa -P | awk '{print $1}' > packages.txt
nix-env -f '<nixpkgs>' -qaP -A haskellPackages | \
    awk '{print "nixos."$1}' >> packages.txt 
#vim: ft=sh
4 Likes
#7

awesome! Time to open a new project on GH, I have a few fixes, say,

nix run nixpkgs.rxvt_unicode -c urxvt -tr -fg white -bg black -e bash -ic "nix run nixpkgs.vim -c vim -R -c 'nnoremap q :q!<CR>' ${DEFPATH}; vimb"

Also, it does search by options, which is nice, would be interesting to search also by config values. For example, make environment.etc. -> to list all the /etc files and where they are set.

2 Likes
#8

Just as a note the “vimb” at the end of the urxvt line was some kind of clipboard typo.
This is a great addition, thanks for sharing

1 Like
#9

I made Deoplete sources for the package and option lists.
It’s unlikely that anybody else would use them but I’ll post the file.
You need to change the file paths in the script and obviously put it in the proper subfolder of your vim-confs.
As an added note, this is basically the first time I’ve touched Python, so there is probably some room for improvement here; but this works.

Ideally this this would be expanded to source more information. Deoplete can easily show things like a preview of the current configuration settings for instance just like the FZF script does, but this’ll do for now.

# ${NVIMCONFS}/rplugin/python3/deoplete/sources/nix.py
import re

from .base import Base
from deoplete.util import (convert2list, parse_file_pattern, set_pattern)

class Source(Base):
    def __init__(self, vim):
        super().__init__(vim)

        self.name = 'nix'
        self.filetypes = ['nix']
        self.min_pattern_length = 0
        self.rank = 500

        self._object_pattern = r'[a-zA-Z_]\w*(?:\(\)?)?'
        self.prefix = ''

    def get_complete_position(self, context):
        m = re.search(self._object_pattern + r'\.\w*$', context['input'])
        if m is None:
            return -1
        self._prefix = re.sub(r'\w*$', '', m.group(0))
        return re.search(r'\w*$', context['input']).start()

    def gather_candidates(self, context):
        # Run Packages
        with open("/etc/nixos/packages.txt") as packs:
            pkgs = [{'word': x, 'menu': 'NxP'} for x in
                    parse_file_pattern(
                        packs,
                        r'(?<=' + re.escape(self._prefix) + r')\w+'
                    )
                    if x != context['complete_str']]
        # Run Options
        with open("/etc/nixos/options.txt") as opts:
            opts = [{'word': x, 'menu': 'NxO'} for x in
                    parse_file_pattern(
                        opts,
                        r'(?<=' + re.escape(self._prefix) + r')\w+'
                    )
                    if x != context['complete_str']]
            return pkgs + opts

Quality of Life Scripts
#10

Might as well post my little script here for searching files in a directory that contains all the given words, no matter what line they’re on. So it’s awesome for writing your own package and searching examples in nixpkgs like "where did someone use makeWrapperArgs and PATH".

#!/usr/bin/env ruby

Dir.glob('**/*') do |path|
  next unless File.file?(path)
  File.open(path, 'r:binary'){|io|
    ARGV.all?{|word|
      io.rewind
      io.grep(/#{word}/).any?
    } && puts(path)
  }
end

Usage like find-all-with-words setupPyBuildFlags buildPython.

1 Like
#11

I’ve modified the script for my usecase, esp. (in italics for what could be interesting in general)

  • launch alacritty instead of urxvt
  • changed the location of the cache files, and made them easier to change
  • handle the case when they are missing
  • only show packages in <nixpkgs> (I have several channels, and every package appears multiple times otherwise)
  • writePacks and writeOpts accept out path and output to stdout by default
  • fewer hardcoded paths

The derivation isn’t perfect yet, e.g. jq and opther dependencies are missing.

You can find the current version at https://github.com/Moredread/nur-packages/tree/master/pkgs/nix-search

1 Like