Nixpkgs CLI Working Group Member Search

Development Nixpkgs Architecture
1

Nixpkgs CLI working group

The Nixpkgs Architecture Team wants to convene a working group for coming up with a CLI tool for Nixpkgs.
This tool would have a similar relationship to Nixpkgs as nixos-option does to NixOS.
Like nixos-option it should live in Nixpkgs so it can be kept in strict sync.

We’ll be organizing this working group with regular hacking sessions, so if you’re interested in joining - let us know! Our first task will be creating a tool to efficiently scrape package metadata from nixpkgs ( more info below ).

We’re looking for a shepherd for this project and additional contributors to get the ball rolling.

Problem

Nixpkgs has tons of domain-specific concepts that are not appropriate to include in Nix proper.
Currently, we have an awkward compromise where commands like nix-shell, nix develop, and nix shell assume and “ad hoc, informally-specified” subset of these concepts, but this is the worst of both worlds:

Possibility

The benefit aren’t just defensive — avoiding the use of Nix commands which constrain Nixpkgs.
Having separate commands also opens the door to making tons of domain-specific functionality more accessible!

  • Far richer meta fields, e.g. with language-specific ecosystem extensions are possible. Can present this information to users much better.
  • Contributer tools, like that for https://github.com/NixOS/rfcs/pull/140, can take advantage of how the code is structured.
  • Smart shells that now how to combine packages (for working on multiple things at once) and other tricks are possible.

Tentative first objective

The “possibility” part is the most fun an exciting.
But it is also open-ended, and that is dangerous for a new working group.
A precise narrow goal can keep things on track, and then momentum can be built for later.

Task

Replace nix-env --query --available --meta invocation in pkgs/top-level/make-tarball.nix with invocation of a new CLI command.

Success criterion

nix-env: Use pname and version attrs in -q by jtojnar · Pull Request #4463 · NixOS/nix · GitHub is no longer needed because the new command handles pname and version correctly without needing Nix to understand them.

Method

Make new CLI residing in Nixpkgs like nixos-option.
Invoke it instead.

Non-goals

It doesn’t need to be stable

At first, this is just an internal tool for pkgs/top-level/make-tarball.nix.
It can be as unstable as we like, since it and its invocation live in the same repo (Nixpkgs).

Tools for humans can come later, and only then do we need to start thinking about stability.

It doesn’t need to avoid unstable internals of Nix

It doesn’t need to use Nix lib as a library, but doing so is explicitly allowed.

There is work to make a new, more stable C interface for Nix to allow using languages other than C++, but we should not wait for it to be ready.
Rather, our new CLI and nixos-option can both serve as additional motivation for it to be finished.

Co-signers

17 Likes
2

Yes! Please.

Related: Rename `nixos-rebuild` to `nixos` · Issue #122715 · NixOS/nixpkgs · GitHub

7 Likes
3

I’d be interested into joining this WG, but I have limited bandwidth for the next month, though, in September, I should be fine.

I have various project for nixpkgs CLI that I am dabbling with, oriented to maintainers, contributors and committers.

2 Likes
4

As author of a tool living in an analogous problem space I’d be interested in joining as well with variable (hopefully: upwards trending) commitment that I would also need to negotiate with other stakeholders.

I have a follow up question/intuition on the initial post above:
It is not mentioning things like nix-init and nixpkgs-review, potentially because of the aim to keep the scope narrow, but if these use cases aren’t completely off-limit, mentioning them might help some people to make the right associations.

EDIT:
Another follow up: How is the relationship with the Nix Team managed and structured (joint sessions, personnel overlap, team-level commitments, etc)? I imagine that this working group would rather sooner or later depend on a excellent collaboration between the two teams to move towards its mission.

3 Likes
5

While not related to a nixpkgs CLI per se, I’ve been toying around with writing a replacement for all of the nixos-* tools in Zig such as this issue describes. Currently, I have plans to unify the following tools:

  • nixos-rebuild
  • nixos-container
  • nixos-enter
  • nixos-generate-config
  • nixos-install
  • nixos-version
  • nixos-help

And possibly other projects like nixos-option and nixos-infect, if possible.

Currently, I have a nixos-rebuild replacement somewhat working, except it does not implement the remote build/target options, and that would take some (by that, I mean a lot) of refactoring. While I have limited time to work on this, I would love to be able to make this happen sometime in the next few months and to open up the work once I think it is ready to show.

6 Likes
6

Does Zig have a good answer to dynamic CLI completion? Rust (i.e. clap) afaik only recently added some support for it, Go had it for a while already via cobra. For extensibility or ingestion of metadata that comes out of the target repo, dynamic CLI completion, eventually aided by a cache, is handy.

2 Likes
7

I do not know of any solution for dynamic CLI completion that I wouldn’t have to implement myself in Zig. However, I’m thinking about using Zig’s build system to generate shell completions from the declarations of commands, so that might come in handy later.

This is still very much a rough idea, though, so it’ll definitely take some work to add features like dynamic CLI completion to it. For right now, I just want to focus on mirroring the vital features of the nixos-* commands and exposing them with a better interface than what we currently have right now, and expand it out from there.

1 Like
8

Sounds awesome. I’d be curious to be able to have a look at the prototype (mainly to understand a familiar programming domain in Zig).

Another element that may be worth considering and/or exploring is a plugin system capability. A plugin system unlocks distributed creativity to tackle new nixpkgs * problems, but also would serve as a sort of graduation academy to stabilize the best ideas into nixos/nixpkgs.

2 Likes
9

Just as a heads up I’d like to avoid any excursions into porting Nix to language X as a prerequisite to making the handful of calls we need to the nix CLI, or existing libnix-util, libnix-cmd, and libnix-expr APIs.

The focus here is to turn out a functional tool that’s easy to maintain and contribute to. I love experimenting with new programming languages as much as any nerd, but upfront I’d like to say “this is not a working group to create FFI bindings or attempt to port core nix routines to other languages”. This is a working group to turn out an efficient and practical CLI tool that will likely recycle some existing Nix C++ routines, the plugin framework, and/or wrapped calls to the nix CLI.

This isn’t to say that I discourage anyone who is interested in ports or rewrites, but I do want to be clear that those activities belong elsewhere.

Let’s hack together a fast and useful tool, and try not to get lost in tangents of high effort ports or cross language linkage. The top priorities are user experience and ease of maintenance. :heart:

6 Likes
10

Not sure if there’s a misunderstanding at play or the statement pertains (partially) to a different context.

From the context I gathered the idea to use Zig is that it can seamlessly import the nix headers & al., while being a saner language to achieve the UX and maintenance goals than X.

In short: avoidance of any FFI requirement, but also avoid CLI wrappers.

But it’s also possible that I misunderstood that aspect.

And there are downsides, too, ofc (popularity, stability, ecosystem maturity). :smile:

2 Likes
11

Yeah nothing personal and it’s not disparage one language vs another, I just want to maintain focus on outcomes.

Context is: many projects fizzle out because they try to rewrite existing interfaces in another language or pick a language that is inaccessible to the majority of developers/contributors.

The goal is to make a CLI tool - there’ll be plenty of brainstorming useful features, design decisions to hash out, fun technical problems to tackle, and architecture/infrastructure to develop along the way. I want to focus energy on those efforts.

5 Likes
12

I think this is definitely a key point and we may need to subsume an ecosystem that goes mainstream with thousands of contributors rather than looking at the status quo assessing this.

Whatever the decision will be, I have no intentions to preempt and am looking forward to this new CLI chapter.

14

So (Towards) stable C bindings for libutil, libexpr by yorickvP · Pull Request #8699 · NixOS/nix · GitHub is meant to go hand-in-hand with Python bindings.

The way I see this WG going down is something like this:

  1. Work with C++ today, because we can immediately get started.

  2. Emphasis on decoupling layer violations in Nix (like the initial nix-env --query --available) before adding new misc functionality.

  3. Maintainability should initially be achieved with less code, rather avoiding C++ if people find it off-putting.

  4. Creating tools like this and nixos-rebuild will help not hurt the case for (Towards) stable C bindings for libutil, libexpr by yorickvP · Pull Request #8699 · NixOS/nix · GitHub; the more florishing tools we have using Nix’s unstable C++ interfaces downstream of Nix itself, the more badly we need a sustainable solution.

  5. Once (Towards) stable C bindings for libutil, libexpr by yorickvP · Pull Request #8699 · NixOS/nix · GitHub and the follow-up Python bindings (which may or may not be in tree) are done, we have the option to rewrite the Nixpkgs CLI from C++ to Python, or something else.

2 Likes
15

So as someone on the Nix team who helped plan this out with @Growpotkin, this WG would be welcome to collaborate us but I also look forward to you asserting your independence! Per my [RFC 0136] A plan to stabilize the new CLI and Flakes incrementally by Ericson2314 · Pull Request #136 · NixOS/rfcs · GitHub I believe Nix itself is better suited to nailing the core functionality / advanced plumbing, and external tools are better suited to nailing the learning journey / misc bonus features. I would love if external tools, like this, would lead in that regard, and Nix proper would follow, providing the foundation atop which the building can be built.

2 Likes
16

While on the subject of CLI and Nix team, could we have a backport of this? restoreMountNamespace(): Restore the original root directory by edolstra · Pull Request #8483 · NixOS/nix · GitHub

It is still hurting a lot of users out there. :wink:

17

Oh my, are we on the Nix Team so AWOL that derailing the thread is the only way to get our attention?! :open_mouth: :slight_smile:

18

Updated to clarify that this Working Group is being recommended by The Nixpkgs Architecture, and that a shepherd/organizer and additional contributors are what we’re hoping to rally for here.

I’ll be contributing but won’t be leading the charge or going it alone on this one :heart:

2 Likes
19

https://github.com/water-sucks/nixos/pull/3

And if the dear reader is furthermore interested:

https://github.com/water-sucks/nixos/pull/4

20

@viperML Wouldn’t you have a ton of input, as well, on this topic?

21

Thanks for the ping, and I wouldn’t mind being involved in the WG.

However, my experience with CLI’s is limited to nh, which just calls nix as a subprocess. It’s written in Rust using clap-derive (which I think is fantastic for CLI’s, see example here) and clap-complete, which generates completions automatically (but I don’t know if it can generate completion dynamically).

Waiting for the bindings is maybe the best bet for writing a tool that “understands” all the metadata.

1 Like