(cross-posted from the mailing list)
Hello!
I’ve just put together a blog post describing some pretty awkward parts (I think) of maintaining and updating nix derivations. I’m particularly focused on cases when the nix maintainer also works on the package itself, and might need to test updates to the source code as well as updates to the derivation, since that’s what I typically do 
http://gfxmonk.net/2018/05/12/a-journey-towards-better-nix-package-development.html
the tl;dr is describing the motivation and basic operation of two utilities I’ve written:
But the post itself describes a pretty long journey I’ve been on to make this process simpler, and issues I’ve run into along the way. If you’ve ever found the process awkward, perhaps you’ll relate and pick up some new ideas. And if you’ve been doing this a long time perhaps you’ll think of different ways to address the problems I’ve run into - maybe there’s something I’ve missed, or a way that nix itself could be modified to make things easier, or a related issue that could be fixed at the same time.
Feedback is of course welcome either here or on the blog.
Cheers,
6 Likes
Thanks for the beautiful blog post, I had a blast reading it. There are so many good tidbits in there, things I guess most maintainers go through. I also had a look at both projects and they make sense.
In your design of nix-pin, it seems like the tool could be maintaining an overlay of the overrides. Was it by design that those can only be activated by using nix-pin as a proxy command?
I have been building something similar to nix-update-source, the fundamental design is pretty much the same. The most painful packaging update experience is when fetching binary packages because they switch on the arch. Do you plan on implementing support for this use-case?
Your observations around maintaining both a default.nix in nixpkgs and upstream make sense as well. It would be great if upstream maintainers could provide a default.nix that can then be re-used in nixpkgs. This would encourage the proliferation of nix a lot. I think we need some sort of escape hatch that allows import-from-derivation is specific circumstances at least.
Most packages are composed out of a src and a main derivation. Splitting the main derivation and the src also makes a lot of sense to me. Instead of:
{ stdenv, someFetcher }:
stdenv.mkDerivation {
name = "foo-1.0.0";
src = someFetcher { .... };
}
If we could standardize what the fetcher returns, it would be possible to inject the src:
{ stdenv, foo-src };
stdenv.mkDerivation {
name = "${foo-src.pname}-${foo-src.version}";
src = foo-src;
meta = foo-src.meta;
}
There are a lot more things to say but that’s the main things.
(sorry for the slow response, discourse didn’t notify me of your post. thanks for the reply!)
Was it by design that those can only be activated by using nix-pin as a proxy command?
Yes, absolutely. Developing with nix can get complex, so I tend to keep my nix install very vanilla (i.e. no custom overlays, very minimal ~/.nixpkgs/config.nix). This way if I nix-build something, I know it’ll probably work anywhere. Making nix-pin explicit helps keep that separation of development vs generally available, and allows me to test both modes by invoking different commands, rather than adding / removing overlays.
The most painful packaging update experience is when fetching binary packages because they switch on the arch. Do you plan on implementing support for this use-case?
I’ve only ever packaged from cross-platform sources, so I haven’t dealt with that. I assume it should be possible, but I don’t know how best to structure it - if you have any ideas about how that should work, I’m all ears
Splitting the main derivation and the src also makes a lot of sense to me.
Funnily enough I ran gup with pretty much the same approach you describe until I started using nix-pin, which made that unnecessary - here’s the removal: gup: simplify build expression · NixOS/nixpkgs@b52f5db · GitHub
I’d be happy for this to be a pattern, but if it’s only a pattern it’s hard to know when you can use it. The benefit of overrideDerivation (used by nix-pin) is that it always works, but it’s quite clunky if you’re writing it manually.
Having said that, I think it’d be possible to add a global nix-pin overlay if people want that, since it already injects itself via an overlay.
One reason not to do that is the increased risk of name collisions - if you have a pin named “foo”, and an argument in any callPackage invocation which is named “foo” but which means something other than that package, passing your foo will probably break something. That should be rare in the recursive dependency tree for things you explicitly build via nix-pin, but becomes more likely if you apply it to everything you ever build.
Regarding nix-pin and overlays, all my projects have their own overlays so it wouldn’t affect me. And in fact it would be handy to have the same facility on the project-level as well.
Regarding the naming collision, I wouldn’t mind if the pinning was mapped to pkgs.<user>.<repo> for example to avoid the name clashes. Or add a .pin prefix.
all my projects have their own overlays so it wouldn’t affect me.
Are all these overlays globally active for your user, or do you activate them on a project / directory level?
The trouble with any prefix is that it then requires source changes wherever you want a pin to be used, which defeats one of the points of nix-pin. If I have a pin called ‘foo’ but it doesn’t get used as a dependency for other packages which depend on ‘foo’ unless I edit them to depend on ‘pins.foo’, I’m back to the hassle of editing files and remembering to revert them before I commit.
You could have it both ways - a globally active overlay which only provided ‘pins.*’ and didn’t do the automatic callPackage injection, but I think it would just confuse people to have two modes.
If I want pins to be used for a given package during development, I use:
(pkgs.nix-pin.api {}).callPackage ./actual-derivation.nix
In my shell.nix, perhaps that would work for you too?