I think we need not more plans, but more experimentation.
It’s hard to reduplicate when things are in different repos, and we just don’t know what sort of abstraction a 3rd repo that NixOS and home-manager should both use should look like. Conversely, with both things in the same repo, we can start deduplicating individual modules and iterative discover what the body of common modules should look it.
Same reason I would want to through NixBSD in the someday and then iteratively deduplicate.
I guess after merging the HM we should then also merge nix-darwin and possibly system-manager?
Yes, good point! This is the first I am learning of system-manager too; that is a nice idea that should be able to be gotten almost “for free” as midpoint between all the others
Why should everything be in a monorepo though? In the 3.5 years I’ve been using nix the nixpkgs repo has grown in size from roughly 19MB to 42MB. But the software I use has not doubled. So effectively I’m wasting more MBs per usage of nix-packaged software than I did a couple of years ago. And it’s not just me, this is the case for every single user of nixpkgs. If we add Home Manager, nix-darwin, system-manager, maybe other stuff, that percentage of wasted bandwidth and storage will only grow. I have really noticed the substantial increase in size in download times which are becoming increasingly more annoying to put up with.
I get the point that it is easier to experiment in a monorepo, but on the other hand it’s nowhere near impossible to experiment across many repos. It might be worth experimenting with a more modular setup of the repositories to make code sharing possible without just putting everything in nixpkgs and exploding its size.
While you’re at it, you might as well merge nixos-hardware into nixpkgs too because why not ;p
As far as I remember, NixOS the operating system was a separate repo, but it lagged behind Nixpkgs the package collection.
Before the creation of overlays and flakes, it was a hard thing to sync them.
Now that they live in the same monorepo, it is way easier to update both a Nix package and its corresponding NixOS module.
Also it is way easier to make integration tests for each of them.
A long time ago I was talking with Okada about splitting NixOS into its own repo, but we never tried to do it.
Nonetheless I still advocate for a monorepo. Filling three PRs for a single package is annoying enough.
Further, NixOS is less than 8% of Nixpkgs monorepo, and the whole HM (plus dotgit) occupies just two NixOSes.
This particular problem will not disappear, since we do not have titanic barriers to include software.
Also, I have vented my complaint with multirepos a short time ago:
Hey, are you stealing my idea? Nops, I steal that idea before
This is somewhat of a tangent, but how code is organised and how its artefacts are deployed are two independent questions. Maintaining code in one file system tree and with one version history simplifies its manipulation a great deal. It doesn’t mean that the result has to be shipped in the same shape. One can automatically split out components into separate units so they can be deployed independently. This requires maintaining clean internal boundaries, and while that’s a bit of effort, it’s a lot less effort than working around stronger but inflexible boundaries, such as separate repositories.
Many years ago I thought that repositories are natural component boundaries and their rigidity helps with maintaining interface discipline. If you dig around you’ll even find me posting about this. But with experience I realised that this completely neglects the value of being able to do sweeping atomic changes, the high cost of learning techniques or building tools to work across separate repositories, and the low cost of automatically producing simpler units of deployment from a unified code base.
In the end it of course doesn’t matter where the code lives. What matters is how much effort it is to work with it for today’s and tomorrow’s maintainers, those people who can really make a difference. The most valuable thing we have is people’s attention and time, and we should optimize for that.
Wow thanks for the numbers! I’ve never thought about the relative sizes before.
Here is a fun thought experiment: even if we decided to split back out NixOS from Nixpkgs, I would still want NixOS, Home-Manager, Nix-Darwin, System-Manager, etc. in the same repo. The “module-using things for system configuration” should be much more tightly integrated with each other than with the bare package set.
While small and insignificant, NixNG (or whatever I rename it too when I get a good idea) would benefit too, many modules are partial copy-pastes from NixOS, the only bits that differ are the actual config attrs. I have thought about pulling in options from NixOS but it would be hard to keep in sync.
i love the idea of a common options file and then each implementation writing their own separate file containing config
some argue that’s duplication but i believe without that you run into too many problems
It looks like the RFC 163 is way fundamental, paving the way to all those projects:
and why not,
any idea how that is progressing? i believe there hasn’t been too much activity lately
Well per what I wrote in Brainstorm for RFC: Assimilate home-manager into Nixpkgs monorepo - #29 by Ericson2314, I don’t think we should wait for 163. Rather, 163 should ask politely we merge some repos first to enable experimentation, and indicate we’re serious about deduplication.
I think that a good way to approach this would be to think about creating some primitives for managing files in home directories (or contested directories more generally where Nix is not the only thing in charge) and then build on top of that.
As others have mentioned there are many modules in home-manager of variable quality so I would be very picky about what to assimilate and what to rewrite.
That’s RFC163 in a nutshell. It’s been stuck for a while, as would be expected for such a major architectural concept.
I think @Ericson2314 is right that waiting for it is a bit pointless; there’s no reason such a change can’t happen after a merge. Meanwhile, a merge can already help get some smaller improvements done, and makes any eventual huge architectural refactor just a tad easier since it won’t be 10 PRs on multiple, very active repositories.
If the only remaining concern is module quality, well, nixpkgs has its fair share of undermaintained code too, it’s not like nixpkgs is overly strict (though it has been improving). Simply having home-manager present in a subdirectory won’t affect the quality of other nixpkgs components anyway, so it doesn’t feel like the impact would be that big.
RFC163 seems to have two parts: 1. generate service configs; and 2. stick them in the right place. Part 1 seems to have gotten all the attention (and doesn’t seem to be making much progress) while I haven’t seen that much discussion about part 2.
So I think we should think more about this second problem and forget about services for the time being. Many home-manager modules don’t use services and simply generate a few configuration files in $XDG_CONFIG_HOME. home-manager currently has a systemd service that links files to the correct place but I’ve been dissatisfied with that for a while… Perhaps there is a nicer way to do this using unionfs/overlayfs mounts on Linux?
Well, if you don’t need to wait RFC 163 in order to inject HM onto the monorepo, then better!
If the idea is to do some incremental migration, then we need some proof-of-concept:
But I am not so versed on that minimal machinery; I need to study this.
@roberth had idea of sharing one of the modules for Nix itself between both projects. I think that alone would be a good and sufficient proof of concept, for me.
https://wiki.nixos.org/wiki/Wrappers_vs._Dotfiles
Personally, I recommend the dotfiles approach (Yes it requires much more work but it’s simpler for the user to use in the end), we could also adopt a hybrid approach where provide dotfiles/modules when we can and the user can use wrappers when there aren’t any.
The nice thing about wrappers is you can easily send part of your closure (i.e. just VIM + vimrc) to another machine without having to deploy the whole home-manager.
TBH I thought wrappers were always the ultimate end-goal for managing home but /shrug