tl;dr: any way to specify projects (e.g., Git repos to clone) declaratively?
Background
I use a couple of different machines for development; one of the great things about NixOS is that they can be as in-sync as possible and moving between them is seamless.
My ultimate goal is:
one-button install on a new machine (minus partitioning, hardware quirks, etc.)
once that’s happened, a git pull && sudo nixos-rebuild switch in my dotfiles gets me totally up-to-date
I’m making good progress toward that, but one sticking point is managing my work projects.
Problem
I have many Git repos I contribute to on a regular basis. Some of them are Nix-ified, but others are not (and will not be anytime soon). I don’t want to have to manually check out new projects. Further, I often want a personal Nix development environment to be inserted in the repo (.git/info/excluded).
My ideal tool:
sets many Git repos at once
bonus for supporting other VCSes
bonus for “can be run from Nix”
nice if it has a pretty UI for seeing all Git repo state
configured declaratively
with extra files (that get .git/info/excluded)
Attempted Solutions
Manage Git in Nix
Currently, I’m using a terrible hack in my home-manager configuration that tries to check out a repo if it doesn’t exist (and copies in my shell.nix and manages the .git/info/exclude file).
This mostly works, but it’s ugly and requires SSH access to the repo (breaking my one-button-push install).
Other tool?
I’m increasingly thinking that there should be some external-to-Nix tool for doing this, and I should manage the configuration for that tool in Nix. Ideally I could call it
There’s a bunch of many-repo-management tools out there (gita, myrepos, repo, etc.) but they don’t seem to quite fit the use case (often not declarative).
Conclusion
How do other folks solve this problem? Something tells me the Nix community has tastes and needs similar to my own.
Hmm, I’m not aware of such a tool. I think it’s a little tricky since git repos themselves are mutable objects… Your “terrible hack” doesn’t sound so terrible to me though
Appreciate the thought! I’d be curious to hear if others would have a use for this; if so, I might look into trying to publish a module or upstream this into home-manager or something.
Maybe the move is to separate the repo configuration from actually using the VCS.
That is, Nix would be responsible for:
ensuring the directories exist/don’t exist
setting up proper remotes (notclone or checkout)
excluded files in their proper place
The above could support many different VCSes, and it wouldn’t matter if e.g. GitHub SSH access wasn’t set up yet.
And we could use another, more imperative tool (git or some wrapper) for the source code state:
checkouts/pushes/pulls (git)
getting status across many repositories (one of the fancier tools)
I’m not aware of any existing tooling here, but you aren’t alone. I’ve never felt like I grasp a full vision here, sorry if this just ends up as a hodgepodge of thoughts…
When I was still fairly new to the ecosystem I made a thread asking whether Nix could support some user data backup/restore concepts. I have a better sense now that it would be masochistic to build this in Nix itself, but I still think this is a hole in the ecosystem, and I do picture a Nix interface.
General needs:
I imagine chunking the main needs like:
describing the filesystem preconditions for using some software (i.e., it requires these files to be present to work; you may not need the files after removing it; the Nix installer could benefit from something like this )
describing data you want to deploy in the same clear, transparent manner you describe the code (i.e., being able to check out an old test server config and re-deploy it as long as the data identifier is still present in the data sink [backups, or S3, or whatever])
describing user data that probably needs to be squishy/mutable
Looking forward?
In December I was reading IRC and had a (potential) lightbulb moment. You can see these IRC logs (one, two) for a smidge more context, but basically: I wonder if it is more feasible to describe and work with stateful storage in Nix if we specified some mountable container, and some path at which to mount it.
This doesn’t really solve the problem, but it marks a clear line where Nix may be able to terminate its ownership/responsiblity in a fairly idempotent way (i.e., Nix deals in whether the mountable container exists or not, and is mounted or not)?
For now:
My own present solution is to back my projects directory up and restore it from my system bootstrap script (this script is mainly focused on macOS for now; I still need to spend some time working out the equivalent nixOS process…).
I don’t really like the lack of granularity, but simplicity wins in the short run.
I don’t really like having to carry some unmodified or fully-pushed repos in my backups just to get this easy restore. I recently posted an SO question about dehydrating/rehydrating git repos with an eye towards figuring out how to declare these and materialize them as needed later.
Add me as a +1. I’m not sure how tied to Nix I want it to be (other than easy to orchestrate declaratively via Nix), but I’ve long imagined a new tool:
a(n optional) daemon that can keep git status information hot
it manages my git checkouts and worktrees based on some declarative manifest
I can instantly know at any time if I have “un-pushed data” anywhere
maybe it even auto-commits to a shadow branch, with auto-pushes, and auto-ff-s in other locations where this tool is running, providing an optimistic auto-sync sort of functionality
In my head, I would never git clone or git worktree ever again, I would interact with my manifest and then have the tool/daemon ensure everything is in place.
I’ve also had this problem, and have pondered the possible solutions (your home-manager solution really isn’t too bad though). I’m planning on giving the services.zfs.autoReplication module a try at some point.
Currently I’m also using impermanence and zfs to keep my stateful setup in isolated zfs datasets. I’m thinking that syncing these datasets (containing my home for example) using said module would work out pretty well. If you are a zfs user, you may want to give something like this a try.
I feel a little less alone now! Thanks all for the input and ideas.
It sounds like we all have slightly different goals, so I’ll try to find the greatest common factor and see if it’s a good fit for upstreaming into home-manager.
My initial gut feeling is that we want either:
some declarative multi-repo tool that we can configure in home-manager (possibly cleaner, but runs into the N+1 problem – maybe some existing solution would be amenable)
a bare-bones home-manager model to clone “dehydrated” repos that integrates well with an existing multi-repo tool (this is much less work)
I’d be willing to shore up my home-manager module and try to make (2) happen, but I probably don’t want to tackle (1). If you all think (2) would be worthwhile, I can sketch out a plan for a well-tested, well-scoped module and propose it over at the HM repo.
Since git objects are immutable, I wonder if they could be kept in the store. Like what if gitnix commit commits into the store? And clone clones objects into the store.
)