Announcing Nix-manipulator (Nima): structured edits for Nix that keep formatting

Announcements
1

(I do not have permissions to post in Announcements, so I am sharing this here. Moderators are welcome to move it.)

nix-manipulator is a Python library and CLI for parsing, transforming, and reconstructing Nix while preserving comments and layout, in line with RFC 166. It is built on Tree-sitter using the nix-community grammar.

Motivations

Nix is often described as “JSON on steroids”. I like this definition but did not find an easy way to update the values in a Nix file without doing string replacements or using regular expressions.

This got me curious about how nix-update proceeds… and it just uses string replacements.

So during SaltSprint 2025 this summer, I decided to tackle this problem and started using tree-sitter-nix to parse Nix code, transform the AST into opinionated and higher level structures and regenerating Nix code while preserving the original format.

This approach does not evaluate the Nix code, so definitions inside let ... expressions or function calls are accessible.

Nix-manipulator is written in Python, with the goal to make this accessible from simple scripts, from a REPL or to be integrated into the existing Python tools.

I started writing a command-line interface named nima to skip launching Python manually, which currently looks like:

nima set --file package.nix doCheck false

Now that I am back from holidays, I finally had time to publish an initial release, which you will find behind the links below.

Status and feedback

An initial release is available.

Not all Nix syntax is supported yet, but 73% of the files in nixpkgs can already be manipulated.

I am looking for feedback on:

  • Real-world edge cases and your use-cases
  • API ergonomics for common edit patterns
  • CLI subcommands and flags that would be most useful

Contributions are also welcome.

Repository: GitHub - hoh/nix-manipulator: Parse, manipulate, and reconstruct Nix source code with high-level abstractions.

PyPI: nix-manipulator · PyPI

56 Likes
2

Ooh, nice, I was just starting to build exactly this Sunday night because I stumbled upon nix-update’s limitations.

I wanted something nice and generic for passthu.updateScripts, but everything out there just seems to use regexes that completely fail if you e.g. use srcs or have an otherwise less straightforward source definition.

Thanks so much!

6 Likes
3

Fantastic work! Would this be applicable for projects like SnowflakeOS, enabling traditional graphical user interfaces for system administration on NixOS?

There was a recent discussion around this on the fediverse in this sub thread, as well as the similar notion of reverting to JSON for a more machine editable markup language:

2 Likes
4

Thank you for your supportive comments!

@ruffsl that is exactly one of my motivations.

I just added some unit tests that demonstrate how to do this, have a look here:

4 Likes
5

Haha, I’ve also been working on the same thing only with JS and tree-sitter-nix so that manipulations can be compiled into a standalone executable (with deno) or run client side in a web browser. Glad to see so much support for the idea!

I’ve got some great usecases (and annoying edge cases) to talk with you guys about.

1. nix-proj add my-pkg

All the nix based project-manager tools (like devbox, and devenv) suffer from the same problem: at some point we need to add complicated custom nix code and custom nix code usually doesn’t integrate well with their very curated system.

Automated editing of a flake.nix could both get the ease-of-use of those systems, while still easily supporting custom nix code. I’ve been working on this in a way that supports nix historical version search, and my recent Xome project.

E.g. nix-proj add grep, or nix-proj home config zsh.enable true

The key is: list literals of module-like elements. For example, instead of having a list literal for buildInputs and another list literal for nativeBuildInputs, we need one list literal inputs, and then the elements have flags like isBuildInput = true; and isNativeBuildInput = true;.

Automated editing of flake inputs could be incredibly nice.

2. Home manager

The home manager module system is already a natural fit. Just auto add modules from the CLI with sanity checks and rollbacks for conflicts.

3. System modules / System config

Not my area of expertise, but I think eventually nix-env style installation should be replaced with auto-editing of a declaritive system. I’d love a GUI app store for nix, but you guys probably know more about how to approach that. All I ask is, please create an automated edit system that supports modules/packages from historical versions of things (e.g. don’t just limit the CLI to whats available on the users active nixpkgs channel)

Edge cases

In terms of edge cases, man I hate with. When I want to do static analysis, like finding what vars are unused, stuff like with lib; always wrecks it or makes it a pain.

So! That is also an opportunity, I’ve always wanted a source-editing tool tool that factors out with and replaces it with the more verbose counterpart (lib.thing or thing) depending on whether or not thing is available in the parent scope.

2 Likes
6

Design

A nix query syntax would be really really really nice. It could compile to the tree sitter query syntax, and basically just be a less verbose version of it.

Ex:

nima
  --find-first '$mkDerivation @{ @any+ ($version @= (@any)#value1 @;) '
  --replace '#value1' ' "1.20.1" '

$ means identifier literal
@{ means start of attrSet literal
@any is kind of like regex’s .+, except its hierarchy aware (iterates at one level)
#name is kind of like a capture group.
(These could match up to the python you’ve written for all the different expressions)

Then a tool like nix-update could easily use this to a best-effort replacement

3 Likes
7

Thanks for sharing these use cases. A JS library would also be useful for Web UIs.

Regarding the with expressions, I would like to add a way to manipulate the code that is transparent to its presence. One should not have to change the manipulating code depending on its presence, but the result should keep it if present.

I would like to implement something similar for the let expressions to offer to manipulate them transparently. Identifiers (variables) can be followed from their usage to their definition and I would like to link those internally.

On the topic of the syntax for editing Nix code, there is some interesting inspiration in XPath (for XML) and JS Query Selectors.

Example: /Wikimedia/projects/project/editions/edition[@language='English']/text()

I am still reflecting on how to elegantly express function call arguments and access to let definitions etc.

8

I wouldn’t overthink it. @jeff-hykin is referring to tree-sitter’s query syntax, which is a perfectly cromulent S-expression based language specifically designed to query tree-sitter-parsed languages.

Why reinvent the wheel? Since the syntax already exists, people know how to use it, and the very library you’re using to parse the language already implements it, so you don’t have to write much code either.

I can see making it a bit more concise with simple substitution like @jeff-hykin suggests, simply to make CLI use more ergonomic, but anything more involved than that is just going to create a lot of empty work and probably make it harder to use.

2 Likes
9

Thanks for pointing out to tree-sitter’s query syntax, I was not aware of it. That syntax does seem pretty advanced and quite hard to read however, and may not be simpler than Python code for many people familiar with the later.

nix-manipulator currently supports the following syntax:

Set a value in a Nix file

nima set --file package.nix version '"1.2.3"'

or

cat package.nix | nima set version '"1.2.3"' > package-updated.nix

Set a boolean value

nima set -f package.nix doCheck true

Remove an attribute

nima rm -f package.nix doCheck

Test/validate that a Nix file can be parsed

nima test -f package.nix

This lacks a way to express that the field doCheck is an argument of the function being called and not a field from a top-level attribute set. It is a simple syntax in this spirit that I would like to support for the command-line.

10

Reminds a bit of ast-grep: GitHub - ast-grep/ast-grep: ⚡A CLI tool for code structural search, lint and rewriting. Written in Rust

Reason I didn’t went for something fancy in nix-update was that this can easily break if you don’t also implement some form of data provenance i.e. people do stuff like:

let
  someVersion = "";
  
in
...
  version = someVersion;
11

I think this kind of query would probably be flexible enough to match that:

((binding 
    attrpath: (identifier) @ name
    expression: (string_expression) @ value)
 (#match? @name "[vV]ersion$"))

Obviously there are pros and cons to this kind of thing, and no flak at all to the current implementation in nix-update. But I do think an implementation of something similar could just clearly state which types of version declarations it’s capable of updating, and expect users to specify their versions like that; any missing reasonable schemes can just be added over time.

Anyway, this kind of tool is useful in general, it’s a faff to have to pull out python and the tree-sitter library just to substitute a few nodes. I was looking for something like ast-grep, was surprised I couldn’t find it, thanks :smiley:

2 Likes
12

Why reinvent the wheel?

@TLATER I think you’re right. After posting I went and wrote a query to append a nix value to a list, and it wasn’t as verbose as I expected.

What I think could work super well I think is action + a query to reduce scope of the action:

# no scope (global)
nima set --file package.nix version '"1.2.3"'

# change "version", only inside python3 = {*here*}
nima set \
    --file package.nix version '"1.2.3"' \
    --within '(binding (attrpath) @key (#match? @key "python3")) @scope'

# change "version" inside python2 = {*here*}
nima set \
    --file package.nix version '"1.2.3"' \
    --within '(binding (attrpath) @key (#match? @key "python2")) @scope'

# those treat "@scope" as a special/required capture (other @names are not special)

Maybe all that is needed is a CLI tool for doing that^ with actions like setAttr, deleteAttr, appendToList, prependToList, concatToList, mergeAttrSet, replaceString, appendComment

2 Likes
13

I completely understand the reasoning and the approach chosen.

Following identifiers from their definition to their usage and in reverse is something I would like to implement, such that the example below would be supported:

{ ... }:
let
   someVersion = "1.2.3";
in
somePackage {
    version = someVersion;
}

There are of course limitations to both approaches, as the following code would fail naïve string replacement:

{ ... }:
let
   dependencyVersion = "1.2.3";
   someVersion = "1.2.3";
in
somePackage {
    version = someVersion;
    ...
}
14

Regarding query syntax, the current implementation of the CLI expects a full path.

You should then use:

# change "version", only inside python3 = {*here*}
nima set python3.version '"1.2.3"'

Result:

{
  version = "untouched";
  python2 = {
    version = "untouched";
    other = "bar";
  };
  python3 = {
    version = "1.2.3;
    other = "foo";
  };
};

I agree that a more advanced query language would be useful, for example to replace version only if other == "bar".

As a Python developer, my first instinct brings me to a syntax in the style of Pandas:

nix set "[other='bar'].version" "1.2.3"

I would however use a different syntax in Python itself, something like:

from nix_manipulator.parser import parse
code = parse(...)
code.filter(other="bar").version = "1.2.3"

I will be at NixCon next week by the way, looking forward to discuss the topic in person.

15

I published a minor update.

This release removes trailing commas after ellipses, adds support for inherit (from) ... and allows passing integers directly to binary expressions.

30948/39574 (78.20%) Nix files from nixpkgs could be reproduced , from 28991 / 39573 (73.26%) Nix in the previous release.

In progress are the support for assert expressions and associating comments to the operator of binary expressions.

8 Likes