Prebuild before applying npins update

Help
1

I’m currently using channels on my system, and would like to switch to something like npins to better manage nixpkgs and other channel versions. As I understand it, the update flow is like this:

  1. npins update—updates sources.json
  2. Rebuild
  3. Revert sources.json if build fails

The trouble is, the rebuild step can take quite a while. (For both the system configuration and various dev shells, I often have patches applied, either to packages or to nixpkgs itself, which means a rebuild after a channel update can take a long time and result in lots of compiling.) During this time, I may find I need to add a new package to the environment, and I’m stuck until the rebuild completes. What I’d like to be able to do is add the package and rebuild the environment using the channel versions in the current generation (fast, since the new package is the only change) while the build using the new versions proceeds in another tab.

Basically, I’d like the flow to look something like this (handwavy, using made up npins subcommands):

  1. npins stage—Like npins update, but saves the new versions in a staging file (never checked in to source control)
  2. Somehow kick off a build using the staged versions.
  3. While that’s running, add a package to *.nix file.
  4. Rebuild and switch with current versions to quickly add the package to environment.
  5. When (2) completes, npins apply-staged
  6. Rebuild and switch to new versions (fast, because the new derivations have already been built).

Has anyone created a workflow like this? Is there a way to accomplish it using existing tools?

Thanks!

2

How about just cloning your repo into a second directory and running your various commands in there, without calling npins update first?

npins update is exactly the same as nix-channel --update, with the exception that the metadata for your channels lives in the same directory as your nix code. If you use the commands the same, there should be no difference in behavior or speed.

Longer-term my advice would be to make fewer downstream patches.

2 Likes
3

There are many reasons a rebuilt after an update might be lengthy (overlays, overrides, site-specific patches, uncached packages, et cetera), so I don’t think “don’t do that” is necessarily a practical solution.

For now, I’m going to experiment with adding an npinsDir arg ({ npinsDir ? ./npins }:) to all of my relevant .nix files. That way, I can do

cp -r npins npins-next
npins -d npins-next update
nix-build shell.nix --arg npinsDir npins-next -o result-next

to prebuild the updated packages, then

rm -rf npins
mv npins-next npins

to apply.

4

Seems like a lot of ceremony to reinvent git worktree (i.e. git worktree add -d ../temp-build), then do whatever in temp-build. If you like it you can turn it into a real branch and commit, if you don’t you can remove it without consequence.

2 Likes
5

That’s not what I said. I’m saying there’s a solution that doesn’t require a custom build process, and that you don’t seem to understand the tools you’re working with. @bme is suggesting another way to do this that is much less overcomplicated.

I do also still insist that most reasons for rebuilds taking longer are borderline anti-patterns. All of the reasons you describe boil down to maintaining software downstream; assuming you’re doing so for a good reason, those patches should be sent upstream and removed downstream eventually, perhaps replaced with configuration. But if you have that many it’s quite likely you’re missing something.

The only legitimate reason long-term IMO is proprietary-but-public builds (i.e., not redisitributable, but not distributed in precompiled form), which in practice is basically just nvidia/cuda, for which a cache exists.

The only other exception are uncached packages, I guess, which only happens if you’re following a -release or -small branch, or when a package fails to build. You shouldn’t be doing the former, and for the latter you might as well cancel the downstream build since it’ll almost certainly fail downstream too.

6

Sure, but my whole reason for switching to npins is to try to have a better user experience than using channels.

Maybe I’m missing something, but using git worktree seems like it would be a lot more ceremony to achieve the same end. First, I have to be using git (as opposed to some other tool) to be managing the files in question, with the repository scoped in a way that the worktree approach makes sense. Then I have to create a the worktree, build in the worktree, make a new branch in the worktree, commit in the worktree, and then finally pull the change back into the main branch.

What I liked about my approach is I can easily create a couple of aliases and just do

npins-try-update
npins-promote
git commit / pijul record / whatever

in a background tab without having to deal with any of that.

Maybe I’m missing something, or maybe my use case is different from yours, but in either case, just telling me I don’t seem to understand the tools without saying why or how isn’t very helpful or welcoming.

I actually have very few overrides, but using an overlay to override just one package to pass one argument to it (such as enabling a feature on the ffmpeg package) can trigger a cascade of rebuilds. And even overriding a single large leaf package can make updates take long enough to make it desirable to be able to make small tweaks to an environment while the update is building.

7

Right, I’ve been assuming you use git. I would recommend git for this, as @bme suggests. Try it out, it’s far less effort than you think, and it avoids having to make code changes so that you can stick to common practices.

If you don’t like being locked into git for VC, any VC tool will allow you to clone your repository, as I suggested in my first comment. git’s worktree is just a slightly more convenient way to do that. Other VC systems will have similar features, too.

Right, yes, extra features on ffmpeg will cause this. One way to deal with this kind of scenario is to make that override only to the package where you need the extra ffmpeg feature, e.g.:

{ pkgs, ... }: {
  environment.systemPackages = [
    (pkgs.mpv-unwrapped.override {
      ffmpeg = pkgs.ffmpeg.override {
        withRtmp = true;
      };
    })
  ];
}

This will limit rebuilds to only when mpv-unwrapped changes, instead of cascading the flag change to a bunch of software where you don’t need it. Using ffmpeg-full instead would also save you the ffmpeg rebuild.

If you don’t actually need the bonus ffmpeg flags when ffmpeg is used as a library, use the ffmpeg-full package straight-up instead. Then you need no rebuilds.

Or if you really want to enable only a specific feature, don’t use an overlay:

{
  environment.systemPackages = [
    (ffmpeg.override {
      withRtmp = true;
    })
  ];
}

In general, I’d recommend avoiding overlays where possible, precisely because they cause cascading changes when most of the time you don’t need that.

But in fairness, I don’t know your use cases. If you showed us exactly what you override why, we could probably help cut down on your build times with suggestions like the above, and solve the root cause.

1 Like
8

Here is what I think: I think what you want to do is work with a fork in the road. That’s what a VCS is for, imo. If you turn something into a function, that isn’t really a function, just so you can parameterise is when you want to test changes to the text that seems to me like solving the wrong problem and it complicates the nix code for no reason, which you carry around everywhere. If you aren’t using git fine, but if you are then there is no such thing as “scoped in a way that works”. worktrees are cheap. you don’t make a new branch in the worktree if you don’t want to: you can do that as part of the worktree creation (in fact the default creates a branch for you with the same name as the tree). commit and merge are also one liners. You don’t want to do that fine, but it’s not a lot of work at all, it’s literally the same number of commands with all added benefits of actually tracking two separate work streams.

I am seeing the problem as

I want from time to time to make mechanical changes (npins update), that might take a while, and might fail. I don’t want this to interrupt what I am doing.

git worktree was made for this. I would honestly write an little script that spun up the worktree and a systemd-run temp service to do the built with low priority cgroup/slice with a libnotify notification to let me know when it was done and an action to launch a term with the logs if it failed. Then I’d get on with whatever else I wanted to do in the mean time. You could go further, you could auto merge with –ff-only if the build succeeds, you could auto rebase and retry if the rebase is clean. Mini local ci.

I honestly like this idea enough that I might implement it for myself (I have nix wrapping buildroot, it takes 40m to build an image).

9

I’ll be curious to see what that looks like if you do!

I don’t think using a git worktree is at all a bad solution if the repository just contains the relevant nix files (though I still find the flow of committing to a branch (or detached head) in the work tree, copying the commit hash, and then cherry picking it into the main branch on the main tree to be a bit tedious). However, when the repository includes more than that (e.g., a larger project that has a shell.nix checked in for development, and I know some folks use git to track their whole home directory (ignoring non-config files)), it can get more unwieldy.

This is part of what I was referring to when I mentioned the repository needing to be at the right scope.

In my case, I run NixOS and like to have my home manager generation inherit the channels from my system configuration (to ensure compatibility with system libraries and to reduce store redundancy). So as not to need root most of the time, (and to match my home-directory-installed Nix setup on my Mac), I use home manager in standalone mode. With channels, my workflow is update, rebuild nixos, rebuild home-manager, and then switch (or rollback).

With npins, I want the workflow to be update the pins in a staging folder rebuild nixos using the new versions, rebuild home-manager using the new versions, and finally, if successful, apply the new versions and switch. That way I’ll be able to tweak both my nixos config and home-manager config while the updates are building.

With my approach, I was thinking I could just pass the location of the staging nixos generation when nix-building the staging home-manager generation. Unless I’m missing something, I wouldn’t be able to use the git worktree approach without switching to the home-manager nixos module, which I’d rather not do.

I realize this means that my home manager config isn’t strictly reproducible on its own, and perhaps having the global config be a git submodule of the home-manager config (or having a standalone npins repo that both have as a submodule) would be more principled, but I was trying to keep things relatively simple and not depend on the particular VCS in use (or occasionally lack thereof).

10

you keep saying this, and it makes me think you’ve never used a worktree before. That’s not how it would work at all. The tree is shared, that’s the whole point. You make a commit, it’s in the same local git repo. git worktree ../foo. cd into foo, do whatever, commit. Delete the worktree. You’ll see in your in your original working copy you have a branch called foo with you commit. do whatever. merge it drop it. There is no copying of hashes unless you want to do that. you don’t have to cherry pick. you’d just merge it like any other branch.

As to your other points. I too have a macbook (work supplied) and so I have a standalone hm, on both my nixos desktop and just hm on the mac (and I agree it’s nice to not root all the time). I have everything in the same repo :upside_down_face: (it’s kind of out of date, but if you are curious see Benjamin Edwards / nixnix · GitLab). If you had two separate repos I could see how it could be annoying, but I’m not seeing it for a single repo unless you are in mono repo territory for something really large. I have 10s of kloc of nix code at work and I have like 4 or 5 worktrees checked out at time with nary a problem.

If you like the idea of managing it with a function, you can certainly do it. I guess suck it and see, if can achieve your workflow and it doesn’t piss you off then who cares what any anyone else thinks :).

1 Like
11

I think you’re missing my point, which is that you have to go back to the main working tree and apply the change from there (whether via merge or cherry-pick or rebase, branch or detached head in the worktree). I have used worktrees plenty, and I do find it tedious if I’m not actually trying to work on two branches in parallel. Is using a branch and a worktree more principled? Probably. But it still feels like more ceremony to me, not less.

Do you have two clones of it, keep it root owned (requiring sudo to edit it), or do you accept that a user-privilege process might some day be able to edit your system configuration? (Admittedly, if an adversary got remote access as my “unprivileged” user, I’d still be pretty f’d.)

12

I just accept the risk of non-root edits for exactly the reason you mention. By the time someone has that level of access it’s over.

Agree to disagree on the worktree stuff :slight_smile:

13

For NixOS and Home Manager, I ended up going a different direction for now, somewhat inspired by Pinning NixOS with npins, this time without flakes for real | piegames.de.

By default, rebuilding the NixOS or Home Manager config will build it using the same channel versions as the current generation. This is done by pointing NIX_PATH at symlinks in /etc or $HOME/.local/state that are managed by the config. To build with a new version (after npins update), I wrote a script that loads the npins (or any other similar attribute set) into NIX_PATH.

1 Like
14

Maybe orthogonal to the discussion, maybe a useful idea – is it reasonable to have per-(modified package) shell.nixs each in their own git repo? This is what I do with anything fiddly and it keeps stuff nice and contained, keeps usable software at known-good pins, and I can upgrade individually as I have need or time, while also not polluting the system image if I don’t always need the package (and therefore not causing the issue of a system upgrade being beholden to some package). I also use this for assets where the outputs are the most important thing, and I want a record of what software was used to generate those outputs.

15

Yes, this is how development on NixOS is intended to be done, and often the only reasonable way. Since NixOS by design avoids offering FHS dirs there is no good way to develop against globally-installed libraries (i.e., the ways that do exist go against NixOS design philosophy, or require forms of virtualization).

But that’s very off-topic.