Source control for shell.nix

Help
1

One of my favorite things about Nix is using shell.nix to manage development environments. I think @zimbatm wrote up a “my perfect dev env” post somewhere about using direnv and nix-shell in tandem, and that’s basically what I’ve been doing. That was a big driver behind me contributing to the Ruby packaging nix functions and Bundix, actually.

What I’ve been finding is that committing my shell.nix files to code repositories has several drawbacks. When I’m contributing to others’ projects, they’re seen as cruft (and I have several .git/info/exclude entries to attest to that).

Even projects I own or am primary on, there’s a different, subtler problem: shell.nix is itself code, and it gets updated over time. If it’s committed with the source code, when changing branches, nix-shell needs to get run all over again. As a for instance, I have one project that had “src = ./.” that didn’t need it, which was a huge improvement once it got taken out, but every branch from before then reintroduces it.

Maybe this is a problem I share with just one other person - if I’d just nix-shell explicitly, I wouldn’t have to deal with the rebuild. (I think there’s also the problem that --add-root doesn’t seem to work for me…)

Which leads me to my question: does anyone have a good practice for managing shell.nix files in Git? I’m contemplating solutions akin to the Github Pages hack of having an unrelated branch for the shell.nix (or maybe an unrelated remote) or a separate repo entirely… Maybe that deserves its own git subcommands or something? If there’s already an approach, though, I’d love to not spend the time.

4 Likes
2

There are a lot of different things here, let me untangle them a bit:

shell.nix code rot

As you note accurately, shell.nix is code and should therefor be tested by CI. One test I tend to add to the suite is to run nix-shell --run "echo OK" and check the exit status.

Unecessary reloads due to src = ./.;

This can be mitigated by using pkgs.mkShell which doesn’t require that parameter, instead of using stdenv.mkDerivation. It also doesn’t need a name attribute which I find usually redundant.

Reload when dependencies change

Changes in dependencies means exiting the nix-shell and re-entering it. With direnv it’s possible to add a watch_file <file> to automate the reloading when one of the related files changes.

Ideally the CI also publishes a binary cache so rebuilds can be minimized. @domenkozar is working on a solution to make it trivial to add binary caches to projects.

Social acceptance of shell.nix in 3rd-party repositories

This is a bit of an uphill battle. If you look into a lot of repositories, some files become acceptable to add, for example the .editorconfig file. Usually because it provides value to the project and people know about it.

Ideally we can get to a place where developers can checkout any project on github and run nix-shell to get a fully working developer environment.

One way to promote social acceptance is to create visibility. The more projects adopt something, the more it becomes the norm. For this effect I am working on https://builtwithnix.org/ which will be a site promoting Nix and encouraging developers to add a badge badge to their project README that links back to the website. The website will explain how to install and use Nix briefly and point to further material.

Did I miss anything?

7 Likes
3

One friction I’ve been dealing with is that --add-root doesn’t work on nix-shell for Nix 2.0. I see that there’s an issue already and someone’s picked that up, so I’m optimistic that it’ll be fixed soon.

The issue with rot is well taken, but the situation gets complex when I discover improvements to make in a shell.nix, and the old version in other branches is different only because it doesn’t have that improvement. As a for instance, I’m now torn about introducing mkShell, since I’ll have to deal with changes to the old version across branches…

I’m really pleased with watch_file, and use nix already watches shell.nix and default.nix for reference.

builtwithnix.org looks really cool! I’ll need to start adding that badge to my projects, and I’ll look forward to the tutorial on getting Nix set up to use shell.nix files. I especially like the “last changed in 1970” footer :slight_smile:

4

@nyarly note that it’s .org, not .com

1 Like
5

Fixed. Thanks, Jerith

6

In that case I would cherry-pick the patch across branches. Usually these are resolved as being the same at merge time so it shouldn’t create any conflicts.

I also tend to avoid long-running branches and to work on multiple things at the same time. It’s not always possible to do that, it depends on the circumstances of the project.

7

That page is a bit old: Last updated 1970-01-01 00:00:01 UTC :wink:

8

An important note here: anything that’s added to a repository introduces a maintenance burden for the maintainers of that repository, so it’s unlikely that maintainers would be willing to add a shell.nix unless they’re already using Nix themselves, either personally or as project infrastructure.

This is the same reason I don’t accept PRs to my JavaScript projects that add TypeScript definitions, for example - I have no intention of maintaining them, so those kind of PRs really just cause annoyance. We probably don’t want Nix to be in a situation where it’s most widely known for annoying PRs :slight_smile:

1 Like
9

Yes, unless you are also a maintainer on the project please don’t send PRs to add shell.nix.

This goes back to the original point that wasn’t really addressed, could there be a system that maintains out-of-source shell.nix files? What are the requirements and how would it look like?

2 Likes
10

That seems like a good idea that I wanted several times in the past already.

You work on some project for a little bit and create a shell.nix, because you do that for every project anyway. But then it just sits there and rots away.
So having a super simple way of saying “upload this shell.nix (and maybe 1-2 other files) for others to find/use” would be awesome, and if i could automate that via something like direnv where it recognizes the project and downloads suitable shell.nix for me would be even cooler.

There are a few ways I can think of, like setting up a repo that contains those files, sharing them via IPFS, or setting up a simple API to communicate with (most likely a combination).

It’s important that we have a robust way of identifying projects, and most likely also to have multiple shell.nix for different branches, personal forks, etc. But for the first version I wouldn’t bother with that too much, maybe give the option to add a comment for the intended purpose and allow everyone to publish their own version would be sufficient.

It’s also dangerous to have automatic uploads, as it might contain things like credentials (I’m guilty of having put them into .nix files in the past for things like private FTP servers).

With that out of the way, I think we should see what kind of UX would be nice to have (I’m using ainix for this hypothetical new tool, combination of the Japanese for Love and Nix):

DWIM usage on a new project

$ ainix .
Detecting project ...
Found `.git/config` with origin of `git@github.com:manveru/yuten.git`
Determining whether someone you subscribe to shared the love for `gitub.com/manveru/yuten`...
1) Found `github.com/zimbatm/ainix/love/github.com/manveru/yuten` with `shell.nix` and `gemset.nix`
2) Found `github.com/manveru/ainix/love/github.com/manveru/yuten` with a `shell.nix`
Which one would you like to use?
1
Downloading to `.cache/nix/ainix/github.com/manveru/yuten` ...
Linking `shell.nix` to ./shell.nix ...
Linking `gemset.nix` to ./gemset.nix ...
All ready, but please read the just downloaded files before use to make sure they don't contain malicious code.

Uploading your stuff

$ ainix push shell.nix gemset.nix
Detecting project ...
Found `.git/config` with origin of `git@github.com:manveru/yuten.git`
Uploading `shell.nix` and `gemset.nix` to `github.com/manveru/ainix/love/github.com/manveru/yuten` ...
Would you like to notify others about this addition? [Y|n]
y
Sending info to ainix.rocks

Discovering other options

$ ainix list .
Detecting project ...
Found `.git/config` with origin of `git@github.com:manveru/yuten.git`
Querying ainix.rocks for options for this project ...
| 2018.05.18 | zimbatm  | shell.nix, gemset.nix              | 
| 2018.05.19 | manveru  | shell.nix                          | 
| 2018.05.19 | cleverca | shell.nix, default.nix, gemset.nix |

Immediately use a specific shell.nix

$ ainix use cleverca .
Detecting project ...
Found `.git/config` with origin of `git@github.com:manveru/yuten.git`
Downloading from `github.com/cleverca/ainix/love/github.com/manveru/yuten` to `.cache/nix/ainix/github.com/manveru/yuten` ...
Linking `shell.nix` to ./shell.nix ...
Linking `gemset.nix` to ./gemset.nix ...
Linking `default.nix` to ./default.nix ...

Maintaining your list of friends to share with.

$ ainix follow cleverca
Adding `cleverca` to your list of trusted users.
You're now subscribed to `zimbatm`, `manveru`, and `cleverca`.

I think I’ll write up a prototype tonight, because that seems like fun and i just registered the ainix.rocks domain anyway.

3 Likes
11

Keep in mind that IPFS is a distribution mechanism, not a storage platform. You’d still want a reliable backing store, to prevent shell.nix files from going AWOL once they stop being seeded.

Otherwise, a semi-automated externally-hosted shell.nix repository sounds like a good idea!

Another case where something like this would be very useful, is for working with packages from language package managers; for example, if you’re using the jpeg-turbo module from NPM, it’s going to expect libjpeg-turbo to be available in your environment, but no such thing is declared in the package metadata.

To support these kind of cases as well, we may want a more generic ‘out-of-band dependency specification’ mechanism than just for shell.nix.

12

I think those dependencies should be handled by the respective language package wrapping mechanism like we do with bundlerEnv at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/ruby-modules/gem-config/default.nix for example.
I’m open for suggestions, but for now I’ll simply write a tool that does the things I wrote above.
I also thought about simply distributing those files via IPFS to make discovery and download via a hash of the repository name for example, simpler, not necessarily storing it in there. But I also won’t worry about IPFS for now, since the last time I used it my network admin got angry for it probing the whole internal network :wink:

13

A tool to find shell.nix files for development would be amazing. One of the biggest barriers to open-source is tedious dev environment setup.

Been meaning to write a blog post on the importance of wrappers. Nix uses wrappers all the time with helpers like makeWrapper and wrapProgram. I will sometimes create a wrapper repo. That’s where I can write a bunch of setup stuff and keep a bunch of notes messy scripts, submodules, and so on without bothering the wrapped repo. Most of my wrapper projects don’t have much worthwhile and they’re messy so I’m a bit reluctant to share them. I also create Gitlab subgroups for them (bit overkill). Examples: https://gitlab.com/jcrben-not-mine/dokku https://gitlab.com/jcrben-not-mine/git https://gitlab.com/jcrben-not-mine/nix

Unfortunately I don’t have any scripts for automating this kind of thing.

14

ainix seems like a really cool idea. Where’s the issues page so that I can request features :slight_smile:

One thing that concerns me: shell.nix is effectively a RCE in this case (although, really it was when it was added to a Git repo… all the more reason not to simply ship them around.) With direnv in place, it can execute before there’s time to review the file.

As a first order mitigation, some kind of signature on the files might be useful? Probably just transparently applied with push. Alternatively, pull the files to a “holding area” for review before inclusion.

15

There’s definitely the issue of security, which is why I plan to make you review the files unless you already trust a specific user (or even then, it shouldn’t be a lot of work compared to writing them yourself).
My idea was to utilize Keybase for sharing the files and utilizing their web of trust, which would solve two issues with one technology, but it also means additional friction for setting up the whole thing, as you’d need to register with Keybase and join a team. We then have a shared folder in that team with the files.
That way, everyone who wants to use ainix within their company for example could start their own team with confidential files, but our files could be publicly available but are signed and you can verify both integrity and authorship.

I’m not 100% sure that’s the way to go yet, since it’d rely on infrastructure we can’t easily replicate in case they fold, but it would make bootstrapping easier and I don’t have to worry about security of the whole chain.

16

Yeah, I’m not sure what you get by leveraging Keybase that would wouldn’t get from using GPG directly. I personally I’m pretty skeptical of Keybase, and I’d much prefer not to get hitched to their wagon.

17

My issue with GPG is usability, I’m not sure how I’d even go about creating a nice UX around it, but if all we want is that you upload signatures for all the files you add, we still need to build a WoT. Keybase solves the bootstrapping issue nicely by letting you build confidence around identity through all the other connections you already have.

That said, I can imagine a hybrid approach might work, since you can easily pull all the PGP keys of your contacts on Keybase and then we use GPG to sign and verify file signatures. That way you can take care of importing/updating keys yourself if you don’t want to use Keybase but for new users it’d be a simple keybase pgp pull to get their contact keys on their machine and start using them.

18

I recognize that I would rather not accept Keybase’s convenience proposition. All I’m saying is that I would like ainix to use GPG itself rather than requiring Keybase, and if I need to wrangle a little to make it work, I accept that as the cost of my curmudgeonliness. :slight_smile:

19

Is there a particular reason to want GPG specifically? My experience is that it’s rarely the right tool for any job, and that its design very much isn’t suitable for the average user to rely on. There’s a plethora of easily integratable cryptographic options nowadays, so I’m not convinced that “how to add GPG to this” is the right premise to begin with.

(I wouldn’t want to contract with a third party like Keybase either, just to install things. Wouldn’t some sort of collaborative review model with trusted parties [like nixpkgs] be a better option than a WoT anyway?)

1 Like
20

I think what you describe is even more generic, it could work for any types of files overlayed over a git repo. That would work quite well if the files aren’t conflicting with the repository files directly.

For conflicting files, maybe there is a way to use the smudge filters in git to avoid committing the changes. That would be more complicated as the patches might have to be adapted for each branches.

Regarding the discovery mechanism, one option would be to establish a convention that the “overlay-something” branch would be a fresh branch containing only the overlay files. For GitHub specifically it would the just be a matter of listing all the project forks and finding those branches.