Documentation of how to set up a Nix channel?

I’d like to set up a (private) channel for the tools we use at work. Is there some documentation for how to achieve this?

In the end I’d like to tell my co-workers to run

$ nix-channel --add <url> work-tools
$ nix-env --install a-work-tool

in order to have a-work-tool available to them (or something almost that simple). Ideally nix-channel --update and nix-env --upgrade should work too.

Are there any resources explaining how to do this somewhere?

2 Likes

I do a similar thing to point at nixpkg’s master.
I’ve found the manual’s introduction helpful to give an overview: Introduction - Nix Reference Manual
It links to the “Channels” chapter which has more details as well.

1 Like

Isn’t that documentation on how to consume a channel?

I’m looking to produce a channel.

A channel is just a folder (or a tarball) with a default.nix that returns an attrset.

OK, please bear with a beginner though.

  • How do I create the folder/tarball? I’m guessing it has to contain both a default.nix and the pre-built binaries, right?
  • Do you know of a (small) example I can take a look at? (Looking at nixpkgs itself is a bit overwhelming.)

@rycee s home-manager is distributed over a channel.

The GitHub repository lives at GitHub - nix-community/home-manager: Manage a user environment using Nix [maintainer=@rycee].

It contains a default.nix at its root. As GitHub provides a tarball for each commit, as well as the “tip” of each branch, and each tag, you can use that tarball as the channel URL.

It as far as I remember the pattern for the name is https://github.com/${user}/${project}/archive/${ref}.tar.gz, where ${ref} is commit SHA, branch or tag.

3 Likes

Ah, thanks. It looks simple enough, though I’m not sure where/how pre-built binaries are provided.

You can use cachix to provide them. Or set up a hydra instance, that can be used as a binary cache as well as far as I understand.

Or you just don’t provide a binary cache and let your users build locally.

Pre-built binaries are not linked to a channel at all.
The only very weak link is that hydra.nixos.org builds binaries and pushes them to cache.nixos.org and when that’s done, it updates the channel.

You can provide a binary cache without a channel.
You can provide a channel without a binary cache.

1 Like

Some documentation about setting up a self hosted binary cache can be found in the wiki:

https://nixos.wiki/wiki/Binary_Cache

2 Likes

All right, so a channel is just an expression and caching pre-built binaries is orthogonal to that.

However, when I’ve added my channel (a tar ball served up from GitHub), and I then add a cachix cache for the pre-built binaries I seem to always build anyways. I was hoping I’d hit the cache and skip the build, why have a cache otherwise?

When looking at the build output I get different checksums when building (using nix-build in git workspace with the default.nix) than when installing (using nix-env -i my-tool), despite the Nix expression being identical i both cases (i.e. identical default.nix). What could cause this unexpected behaviour?

A common issue can be failing to pin the nixpkgs version. Specifically nix-env uses a different source than nix-build under some configurations. This is the difference between ~/.nix-defexpr and NIX_PATH and channels and root channels and probably a few other settings. A pinned nixpkgs (niv is a good tool for this and possibly the new flakes construct) can help. Once everything is pinned right, builds should give you cache hits.

Other tools that may help, nix-diff and “nix why-depends”

Good point! I’m sorry; I missed part of your original need. The main documentation provides some help but I’ve also gotten stuck trying to build my own channel.

1 Like

There’s a tool called nix-diff that will tell you the difference between the two.

@tomberek I’ll have to look into those tools, but I don’t think that pinning really explains the cache misses since what I do is:

  1. Run nix-env -i pkg produces hash abc.
  2. Push build with hash abc to cachix. (I can’t really inspect what’s in cachix, but the used space increases.)
  3. Uninstall pkg.
  4. Run nix-env -i pkg again to install.

At 4 I expect the cache to be used, but instead it builds, and this build has the exact same hash (abc) as the previous build.

I’ll try to put together a small example.

So, here’s a tiny example: https://github.com/magthe/nix-channel-app

If I build it 5 times I get

$ for i in $(seq 5); do nix-build 2> /dev/null ; done                                                                                                                            <<<
/nix/store/1nbj7d1sh2q8fcrq18n104i3al85132p-nix-channel-app-0.0.1
/nix/store/bc0cnpakr7iqdgxmrvc7a85d53wrrck1-nix-channel-app-0.0.1
/nix/store/0n6ijph1w4q897g7qjv4wgim7ym84kqz-nix-channel-app-0.0.1
/nix/store/0qsz01ixa7kljf0rxrfp2s2apfji3vl5-nix-channel-app-0.0.1
/nix/store/fpzsb595q0pkycil85nvg3f1fmlgfbk5-nix-channel-app-0.0.1

When running nix-diff on two of the derivations I’m told that

- 1s95ggixwbnmb5vnqqsc46s221xj5p8c-nix-channel-app-0.0.1.drv:{out}
+ 1c41qn761fhm9kwjh5arnkn20wcp40nm-nix-channel-app-0.0.1.drv:{out}
• The set of input sources do not match:
    - /nix/store/sx4al9mnidsj379w2mlxw91ghcywxfic-nix-0
    + /nix/store/f2bqsww4i751irz5kibnm20r7n9j4lzd-nix-0
• The environments do not match:
    src=''
        /nix/store/→f2bq→s←x←→ww→4←al9mn←i←dsj3←7←9w2mlxw9←→5→1←ghcywxf←i←c←→rz5kibnm20r7n9j4lzd→-nix-0
    ''

Clearly nothing relevant has changed between these builds, so why would the environments (and thus the hashes) differ?

Doing it that way will include the result link in your next buildinput.

If you remove it after build, you’ll see the same output:

$ for i in $(seq 5); do nix-build 2> /dev/null; rm ./result ; done
/nix/store/cr12612ri4sf54qk42676m810i0bqvkh-nix-channel-app-0.0.1
/nix/store/cr12612ri4sf54qk42676m810i0bqvkh-nix-channel-app-0.0.1
/nix/store/cr12612ri4sf54qk42676m810i0bqvkh-nix-channel-app-0.0.1
/nix/store/cr12612ri4sf54qk42676m810i0bqvkh-nix-channel-app-0.0.1
/nix/store/cr12612ri4sf54qk42676m810i0bqvkh-nix-channel-app-0.0.1

There is also a flag to nix-build that will not even create the result link, but I can’t remember which one it was… I think --no-result-link or something like that.

As Noobz pointed out, it seems your source is changing.

There are some functions that filter out gitignore and result links: https://github.com/NixOS/nixpkgs/blob/e2b4abe3c8f2e09adfc6a52007841d1b96c89371/lib/sources.nix#L36

Thanks @NobbZ and @tomberek!

The change in this commit makes the build behave as expected:

for i in $(seq 5); do nix-build 2> /dev/null ; done
/nix/store/nf608j8d0jpgs7ds1wralmn72shs5jja-nix-channel-app-0.0.1
/nix/store/nf608j8d0jpgs7ds1wralmn72shs5jja-nix-channel-app-0.0.1
/nix/store/nf608j8d0jpgs7ds1wralmn72shs5jja-nix-channel-app-0.0.1
/nix/store/nf608j8d0jpgs7ds1wralmn72shs5jja-nix-channel-app-0.0.1
/nix/store/nf608j8d0jpgs7ds1wralmn72shs5jja-nix-channel-app-0.0.1

Now on to the next issue, how do I get the built version “into a channel”?

I’ve built it twice on Travis, and the second build on Travis ends with:

these paths will be fetched (0.18 MiB download, 0.67 MiB unpacked):

  /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1

copying path '/nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1' from 'https://magthe-temp.cachix.org'...

All done.

So I think that means the cache does contains a built version.

I’ve added the following line to .nix-channels:

https://github.com/magthe/nix-channel-app/archive/master.tar.gz nix-channel-app

and updated channels:

$ nix-channel --update
these derivations will be built:
  /nix/store/xr34rr965qp63fsrmy40b4np3dvgi8yx-nix-channel-app.drv
building '/nix/store/xr34rr965qp63fsrmy40b4np3dvgi8yx-nix-channel-app.drv'...
unpacking channels...
created 3 symlinks in user environment

However, when I try to install the package, using nix-env -i nix-channel-app I end up building the app. The build output (truncated):

nix-env -i nix-channel-app
unpacking 'https://github.com/NixOS/nixpkgs-channels/archive/0c960262d159d3a884dadc3d4e4b131557dad116.tar.gz'...
copying path '/nix/store/2l5bkv1s788nfzzghwf9ckkc6j0zy75k-busybox-1.31.1-x86_64-unknown-linux-musl' from 'https://cache.nixos.org'...
copying path '/nix/store/jc2jzf3ww11696jrp6l20crbzdnj7x5g-apr-1.7.0' from 'https://cache.nixos.org'...
copying path '/nix/store/1z8wd00d9px0n20jzys93rdpafrndqij-aws-c-common-0.3.11' from 'https://cache.nixos.org'...

....

installing
Installing executable ze-app in /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1/bin
Warning: The directory
/nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1/bin is not
in the system search path.
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1
shrinking /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1/bin/ze-app
strip is /nix/store/7bhi29ainf5rjrk7k7wyhndyskzyhsxh-binutils-2.31.1/bin/strip
stripping (with command strip and flags -S) in /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1/bin
patching script interpreter paths in /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1
checking for references to /build/ in /nix/store/64ji2j8160idj14vxm1vfw1ykm0ds0r8-nix-channel-app-0.0.1...
building '/nix/store/kmhhapfv7p8rdwqvjjqinyfrf0fx1w5a-user-environment.drv'...
created 23 symlinks in user environment

It certainly looks like the built binary is identical to the cached version, the same hash, but why isn’t it just pulled out of the cache when installing?

Have you enabled the cachix on that host?