Handy scripts for fuzzy searching nixpkgs and nixos-options


#1

Hey y’all, I cooked up a few little scripts to help make searching for package names and options a bit easier; I thought I would share the results in case it was useful to others, or to get feedback.

This script can be adapted for a ton of other uses, but here is how I have it wired together:
an XMonad hotkey launches a utrvt terminal, running fzf, this provides fuzzy searching on packages or options, using an fzf preview window to see additional information (current settings or package descriptions).
Here are the results:


First we need to generate 2 text files full of all the system’s packages and options:

  1. Packages are real easy, simply do nix-env -qa -P | awk '{print $1}' > allpackages.txt
  2. Options are a bit harder, I eventually made a short haskell program to wrap around the “nixos-option” command to dump every possible option name, I am sure there is a much better solution to this. But here is my script:
{-# LANGUAGE OverloadedStrings #-}

module Main where

import qualified Data.Text.Lazy as T
import Data.Text.Lazy (Text)
import qualified Data.Text.Lazy.IO as TIO
import Data.Tree
import System.IO
import System.Process
import Control.Monad
import Data.Monoid

-- Builds option list from local nixpkgs (including new modules)

nixQuery :: Text -> IO Text
nixQuery q = withFile "/dev/null" WriteMode (\null -> do
  (_, Just hout, _, _) <- createProcess (proc "nixos-option" [T.unpack q])
    { std_out = CreatePipe, std_err = UseHandle null }
  TIO.hGetContents hout)

nixBranch :: Text -> IO (Text, [Text])
nixBranch q = do
  ls <- nixQuery q
  if T.isPrefixOf "Value:" ls then return (q, []) else
    let ls'  = T.lines ls
        lsd  = if T.null q then ls' else map ((q<>) . T.cons '.') ls'
    in return (q, lsd)

mkTree :: Text -> IO (Tree Text)
mkTree = unfoldTreeM nixBranch

leaves :: Tree a -> [a]
leaves (Node x xs) | null xs   = [x]
                   | otherwise = concatMap leaves xs

options :: Text -> IO [Text]
options q = leaves <$> mkTree q

main = TIO.writeFile "alloptions.txt" . T.unlines =<< options ""

even looking at that script again I see a few things that would make it significantly faster. Piping directly to the file instead of using “writeFile” would be a big improvement. But in any case this is functional. Compiling with “-threaded” helps a bit, but full disclosure: this is slow as hell and needs to be rebuilt from the ground up. When I set out I had imagined I was going to to hold onto this tree structure, but in actuallity it shouldn’t construct a tree at all. Instead it should just crawl through the options and immediately write them to the text file when it reaches a dead end. IN ANY CASE IT WORKS haha

Edit: I have made a solution with Nix that is a million times faster. It can be found here for anybody who wants to use it.

to run fzf I use:
packages

cat allpackages.txt | fzf \
    --reverse \
    --prompt="NixOS Packages> " \
    --preview="echo -e \"{1}\n\"; nix-env -qa --description -A {1]" \
    --preview-window=wrap

and for options

cat alloptions.txt | fzf \
    --reverse \
    --prompt="NixOS Options> " \
    --preview="echo -e \"{1}\n\"; nixos-option {1}" \
    --preview-window=wrap

for folks who want to wrap all of is a urxvt launcher:
urxvt -e zsh -ic 'cat allfoo.txt | fzf ...'
Just remember to add extra escapes and what not where necessary.

Extensions:

  1. Obviously improving the haskell program
  2. Add fzf actions to open files that define packages and options.
  3. Colorize/otherwise improve fzf previews

#2

Cool. Your using Haskell to do this reminded me of how nixpkgs-update looks up the attr path given a package name. It also is pretty slow, I bet I could speed it up by caching the entire list of packages into a HashMap, like how you save them to allpackages.txt.


#3

Thank Ryan. Last night I caved and gutted the a bunch of the documentation code out of <nixpkgs/nixos/docs/manual> to accomplish this same task, which runs miiiiiles faster. It has the added advantage of dumping descriptions and other info as well so I don’t have to use nixos-options with fzf (which was a little sluggish).
I do prefer writing scripts in Haskell though, Nix is great for packages but it can be so frustrating when you want to borrow a function thats floating around some .nix file’s “let” section.
Nixly Solution


#4

Your screenshots look seductive. FZF is much more interesting than I expected. I imagine combined dictionary:

  • options (current value, default, defintion file, declaration file)
  • packages (description, defintion file, latest version, current version)
  • tests (defintion file)
  • filenames in nixpkgs source tree
  • … maybe some other runtime information (for current system)? …

#5

That sounds like a solid plan. Be sure to check out my other post where I have a vastly improved nix script for pulling the options list.
And yeah I totally agree, FZF is such an awesome tool. Over the years I always find new uses for it; I know it’s basically just fancy AG, but the interface is just so nice :slight_smile:


Franken-Script to generate NixOS Options docs : with custom modules
#6

I’m back with updates! :blush:

I have added dedicated scripts to dump package/option information, which automatically updates if it is older than the most recent NixOS build.
FZF launches a new terminal window, with the package/option definition in a read only Vim buffer when the user hits enter.

nix-search

#! /usr/bin/env bash

# See if options and packages are outdated
if [[ $1 == "-o" ]]; then
    TARGET="/etc/nixos/bin/options.txt"
    PR="NixOS Options> "
    PREVIEW="nixos-option"
    function seeDef () {
      nixos-option "${1}" | tail -n 2 | head -n 1 | cut -d "\"" -f 2
    }
else
    TARGET="/etc/nixos/bin/packages.txt"
    PR="Nixpkgs> "
    PREVIEW="nix-env -qa --description -A"
    function seeDef () {
      nix-env -qa --json -A "${1}" \
        | jq ".\"${1}\".meta.position" \
        | cut -d "\"" -f 2 | cut -d ":" -f 1
    }
fi

SELECTION=`cat ${TARGET}                                \
  | fzf --prompt="${PR}" --reverse                      \
        --preview-window=wrap:70%                     \
        --preview="echo -e \"{1}\n\"; ${PREVIEW} {1}"`

DEFPATH=`seeDef ${SELECTION}`
urxvtc -tr -sh 8 -e zsh -ic "vim -R -c 'nnoremap q :q!<CR>' ${DEFPATH}; zsh"

# Do updates to package/options lists
NIXAGE=`date -r /etc/nixos/common.nix +%s`
OPTSAGE=`date -r /etc/nixos/bin/options.txt +%s`
PACKSAGE=`date -r /etc/nixos/bin/packages.txt +%s`

if [[ ( $(( ($NIXAGE - $OPTSAGE) / 60)) -gt 5) ]];
then ./writeOpts & fi

if [[ ( $(( ($NIXAGE - $PACKSAGE) / 60)) -gt 5) ]];
then ./writePacks & fi

#vim: sh

writeOpts

#! /usr/bin/env bash

echo $(nix-instantiate - --eval --show-trace <<EOF

with import <nixpkgs/nixos> { };
let
    extraSources = [];
    lib = pkgs.lib;

    optionsListVisible =
        lib.filter (opt: opt.visible && !opt.internal)
        (lib.optionAttrSetToDocList options);

    # Replace functions by the string <function>
    substFunction = x:
        if builtins.isAttrs x then lib.mapAttrs (name: substFunction) x
        else if builtins.isList x then map substFunction x
        else if lib.isFunction x then "<function>"
        else if isPath x then toString x
        else x;

    isPath = x: (builtins.typeOf x) == "path";

    optionsListDesc = lib.flip map optionsListVisible (opt: opt // {
        description = let attempt = builtins.tryEval opt.description; in
            if attempt.success then attempt.value else "N/A";
        declarations = map stripAnyPrefixes opt.declarations;
        }
        // lib.optionalAttrs (opt ? example) {
            example = substFunction opt.example; }
        // lib.optionalAttrs (opt ? default) {
            default = substFunction opt.default; }
        // lib.optionalAttrs (opt ? type) {
            type = substFunction opt.type; }
        // lib.optionalAttrs
            (opt ? relatedPackages && opt.relatedPackages != []) {
                relatedPackages = genRelatedPackages opt.relatedPackages; }
        );

    genRelatedPackages = packages: let
        unpack = p: if lib.isString p then { name = p; }
                    else if lib.isList p then { path = p; }
                    else p;
        describe = args: let
            title = args.title or null;
            name = args.name or (lib.concatStringsSep "." args.path);
            path = args.path or [ args.name ];
            package = args.package or (lib.attrByPath path (throw
                "Invalid package attribute path '\${toString path}'") pkgs);
            in "<listitem>"
            + "<para><literal>\${lib.optionalString (title != null)
                "\${title} aka "}pkgs.\${name} (\${package.meta.name})</literal>"
            + lib.optionalString (!package.meta.available)
                " <emphasis>[UNAVAILABLE]</emphasis>"
            + ": \${package.meta.description or "???"}.</para>"
            + lib.optionalString (args ? comment)
                "\n<para>\${args.comment}</para>"
            + lib.optionalString (package.meta ? longDescription)
                "\n<programlisting>\${package.meta.longDescription}"
                + "</programlisting>"
            + "</listitem>";
        in "<itemizedlist>\${lib.concatStringsSep "\n" (map (p:
            describe (unpack p)) packages)}</itemizedlist>";

    optionLess = a: b:
        let
        ise = lib.hasPrefix "enable";
        isp = lib.hasPrefix "package";
        cmp = lib.splitByAndCompare ise lib.compare
            (lib.splitByAndCompare isp lib.compare lib.compare);
        in lib.compareLists cmp a.loc b.loc < 0;


    prefixesToStrip = map (p: "\${toString p}/") ([ ../../.. ] ++ extraSources);
    stripAnyPrefixes = lib.flip (lib.fold lib.removePrefix) prefixesToStrip;

###############################################################################

    # This is the REAL meat of what we were after.
    # Output this however you want.
    optionsList = lib.sort optionLess optionsListDesc;

    optionsJSON = builtins.unsafeDiscardStringContext (builtins.toJSON
        (builtins.listToAttrs (map (o: {
            name = o.name;
            value = removeAttrs o [
            # Select the fields you want to drop here:
                "name" "visible" "internal" "loc" "readOnly" ];
        }) optionsList)));

in optionsJSON
EOF
) | sed 's/\\\\/\\/g' | sed 's/\\"/"/g' | head -c -2 | tail -c +2 | \
    jq 'keys | .[]' | sed 's/^"\(.*\)"$/\1/g' > options.txt

# vim: ft=sh

writePacks

#! /usr/bin/env bash
nix-env -qa -P | awk '{print $1}' > packages.txt
nix-env -f '<nixpkgs>' -qaP -A haskellPackages | \
    awk '{print "nixos."$1}' >> packages.txt 
#vim: ft=sh

#7

awesome! Time to open a new project on GH, I have a few fixes, say,

nix run nixpkgs.rxvt_unicode -c urxvt -tr -fg white -bg black -e bash -ic "nix run nixpkgs.vim -c vim -R -c 'nnoremap q :q!<CR>' ${DEFPATH}; vimb"

Also, it does search by options, which is nice, would be interesting to search also by config values. For example, make environment.etc. -> to list all the /etc files and where they are set.


#8

Just as a note the “vimb” at the end of the urxvt line was some kind of clipboard typo.
This is a great addition, thanks for sharing