Automatically generating deb packages from nixpkgs

Help
1

I’m talking with someone who develops a lot of audio software about making a system for automatically building .deb files from nixpkgs. They currently have a very large debian and ubuntu repository (156 packages) that makes it easy for people to get audio production software through their package manager as long as they’re on debian or ubuntu. Recently, this developer mentioned they wish their packages could be used by people on other distros like arch. I agree with this, as it would also mean that people don’t have to make duplicate packages of every audio program for every distro.

I thought nix could have a solution for this, and I found this wiki page on how to export a deb package from a nixpkg, but the method demonstrated seems to have some limitations. For example, the exported deb file doesn’t automatically have the names of dependencies, which need to be named as they appear in the ubuntu repositories (“libc6 (>= 2.27)”), rather than the nix names (“clib”).

Is this something that might be possible to do with nix? Are there any examples of similar things being done in the wild?

3 Likes
2

I’m confused. Do you want to convert packages in nixpkgs into .debs? Or do you want to use deb packages as a starting point to make nix packages? Or do you want to build deb packages from debian systems, just with nix as an orchestration tool? Different parts of your post seem to support different conclusions.

If you want to leverage nix to make programs that can run on linux generally, you might want to check out nix bundle.

2 Likes
3

This software ‘sounds’ interesting , what is it?

Building it and adding to nixpkgs might be a good idea.

If they can be build under nix, then something like GitHub - DavHau/nix-portable: Nix - Static, Permissionless, Installation-free, Pre-configured
might be a good idea too.

but nix bundle is probably what you need.

This GitHub - musnix/musnix: Real-time audio in NixOS may also be of interest to them, it gives you all the low latency patches, and all the cool stuff you need for real time audio.

4

We basically want to convert nixpkgs to .debs. Ideally, we want to use the build instructions and resources from nixpkgs to build .deb packages to populate a repository of packages that can be installed on debian or ubuntu, with the goal being to eventually add support for other distros like Arch.

nix bundle looks like it could be a solution worth looking into, but I don’t know if it will cover for everything we need to do. For example, many audio packages are “plugins”, which come in a variety of formats, like VST3, LV2, CLAP, to name a few. I haven’t tried nix bundle yet (I will soon), but I think this could be a difficult thing to solve, as I think plugins act a bit like libraries (they are all .so’s), and they require a host (like Ardour or Carla) to run.

I haven’t asked yet, but I suspect that the developer will reject the idea of making nixpkgs and asking people to install them through nix, as the reason the repo exists is to make it packages more accessible to people through their package manager. Ideally we wouldn’t replace the current repository with something else, as it’s something that many people currently depend on.

5

I see better what you’re aiming for now, and it sounds like nix would have a bit of trouble filling this role. packages built with nix are built to run out of /nix/store, at every level this assumption is hardcoded into everything. To the point that changing it would basically mean starting from scratch. So any attempt to turn nix packages into packages on another system must plan to somehow have a proper /nix/store in place.

nix bundle solves this with user namespace support, and you might be able to make reasonable .debs that use the same method, but it would also probably tend to have some interesting integration issues, such as not loading things from the default directories due to the nix package for the thing having been reworked to run out of the nix store and find its stuff there.

The nix ecosystem as a whole rejects well-known paths as much as possible, to the point that you typically don’t install plugins for other programs by installing a separate package which puts files in an expected location. Instead, such programs often have a function in nixpkgs included with them to produce a version of the package with the given plugins in it’s search path via a self-contained mechanism. This approach wouldn’t blend well with plugin installation in an environment like apt.

6

:joy: I appreciate the pun. Some examples of the software we want to package are

  • lsp-plugins
  • carla
  • distrho

(all these names are in nixpkgs)

Using nix portable is a possibility I have considered, and it’s similar to the idea of using nix bundle ideally we would like to make packages that can go in repos for other distros. I wonder if using nix bundle to generate packages and then putting them in .debs could work :thinking: (I will try all of these as soon as I can).

7

What I find interesting is the example on the wiki page I mentioned:

let
  pkgs = (import ./default.nix {});
  vm    = pkgs.vmTools.diskImageFuns.ubuntu2204x86_64 {};
  args = {
    diskImage = vm;
    diskImageFormat = "qcow2";
    src       = pkgs.ncurses.src;
    name      = "ncurses-deb";
    buildInputs = [];
    meta.description = "No descr";
  };
in pkgs.releaseTools.debBuild args

I ran something similar to this as nix-build ./deb.nix in the root of nixpkgs, and I got a deb for ncurses as the output, but it suffered from the deficiencies I mentioned in the first post.

8

Give nix bundle a try… you may have more success.

2 Likes
9

I’ll post an update when I get a chance to try nix bundle, it seems like it could work.

10

I think the way to look at this is that the .deb files will simply be for “transporting” the binaries to a !NixOS system. You are not going to be able to integrate with any of the existing libraries or binaries on the host, so every application becomes a self-contained bundle.

nix bundle was mentioned, but how about simply creating containers or flatpaks or appimages or one of the other many technologies that solve exactly this problem?

11

Thank you all for your suggestions. nix bundle sadly doesn’t work with plugins. I tried packaging lsp-plugins with nix bundle nixpkgs#lsp-plugins, and it produced a binary that extracted some files to /tmp and tried to run a binary that doesn’t exist. Because plugins function more as libraries than executables, I don’t think nix bundle is the right tool for this. They also need to be placed in a location that a plugin host will look for them in, like /usr/lib/lv2, or another place in the LV2_PATH environment variable.

The reason flatpaks don’t work in this case is that flatpaks run everything in an isolated environment, which prevents plugin hosts from accessing plugins that aren’t in the flatpak unless they are added to the flatpak itself, which isn’t feasible given how many plugins exist (we can’t put every plugin in a flatpak), and how they’re supposed to work (plugins provide flexibility by letting you use the same plugins in any other plugin host). I would love to use flatpak, but it just isn’t right for this use.

The example on the wiki seems to get us most of the way to the result. The specific problems with the result generated by the example on the wiki that the developer pointed out to me are

  1. If you open the deb as a tar and look at its contents, the debian/control file has an empty “Depends:” field
  2. The generated deb is missing a md5sums file

The “Depends” field is set by the debRequires argument in pkgs.releaseTools.debBuild, so I think all we need to do is find a way to get consistent names from nixpkgs to the systems we want to build for (probably through translation), and put them in the debRequires field, and then figure out how to generate the md5sums file and add it to the package. A lot of audio packages have similar dependencies, so I don’t think making something to translate nixpkgs names to debian names would be too much to do.

I’ve got to admit, I’m a bit out of my packaging comfort zone here. It could be that this last step of translating dependency names is just too much, but it seems so close to working.

Like I was saying, if this could work, it would be huge. Not only for nixpkgs (we would likely get a lot of new audio packages), but it would also take a significant burden off of all the maintainers who have to build these packages, as well as it would make a lot of software accessible to more users.

Any other ideas for how to do this are be appreciated. I wish we could use things like flatpak or appimage, but all of them fall short for plugins, which don’t run as executables.

Again, thank you all for your ideas so far, even if nix bundle hasn’t panned out like we hoped

12

nix bundle sadly doesn’t work with plugins.

Well, technically it does - the problem is that none of the software knows where to find them.

The reason flatpaks don’t work in this case is that flatpaks run everything in an isolated environment, which prevents plugin hosts from accessing plugins that aren’t in the flatpak unless they are added to the flatpak itself

And this is very similar to the situation with nix.

Maybe if you can share some details concerning why you want to do this with nix, we would be able to help you.

1 Like
13

A closer look at that wiki post you’re linking says it’s almost not using nixpkgs at all. It’s just grabbing the source that would be used by the nixpkgs build, and then running a build process on a non-nixos virtual machine, using non-nixpkgs dependencies and build systems, without paying any attention to anything in nixpkgs about how to actually build the software. You might as well take nixpkgs out of the equation, unless orchestrating things with nix is easy due to your other infrastructure.

As I originally said, the fundamental assumptions of nix and its entire ecosystem seriously hamper interoperation with apt-installed packages. You have to essentially throw all of nixpkgs out the window and start over if you want the resulting debs not to be a nix container in deb clothing.

1 Like
14

Hmm okay, maybe nix isn’t the right tool for this. It feels like it’s really close to being the right tool (it can export a deb with relatively little input), but maybe it requires more work it seems. If nix isn’t the right tool, I don’t know what would be the right tool is.

I thought nix would be a good tool because it provides a simple way to make a deb package from some common bits of nixpkgs, but maybe it’s not quite enough. I thought that it might be able to do this given how flexible nix’s ability to output other things from its input instructions is. For example, its ability to make different custom disk images of an OS configuration, how it can build things, bundle things, and probably lots of other things I haven’t learned about yet, all without changing input instructions, is super impressive. I guess maybe I’m stretching its features a bit far here.

15

All that flexibility is predicated on the ability to always deploy software in the same filesystem locations under /nix/store, in some form or other. It also generally depends on tightly coupled programs (such as programs and their plugins) being installed and coupled entirely through nix. Interoperation with the rest of the system tends to be a problem (see nixGL for an example of the kind of problem I’m talking about and how we solve it).

16

You may want something more like

or GitHub - goreleaser/nfpm: nFPM is Not FPM - a simple deb, rpm, apk and arch linux packager written in Go

1 Like
17

Those are good suggestions, thanks @adamcstephens! I wish I knew about these sooner!

18

To be honest I don’t see why nix could not be (ab)used to do something along this line. Of course if we want the full purity of nix, nix-bundle or alike can’t be avoided… but they will not use the system’s libraries which defeat the purpose of packaging. But I can imagine other solutions.

  • first, why can’t we simply “unpatch” a nix output? We could reset the loader to use /lib, remove the RPATH, remove the wrappers etc… Also, we could use passthru to provive automatically the name of the dependency libraries (like mylib.debian.packages could provide the name of the debian packages to install to install mylib), and/or use a database to know which deb provides which .so file. I guess some quite specific program might fail if they rely too heavily on patching… but I guess for most programs it does make sense.

Otherwise, the existing deb packaging functions might already be good enough, I guess they use $out=/ to install it in a debian-like systen? If the ondy problem is to automatically provide dependency names, passthru should work nicely.

1 Like
19

My intention for “nix bundle” is to support this use case. There are a few possible conversions to .deb possible; using native deb dependencies, using only nixpkgs, conversions to other portable formats, rewriting store paths, etc. And while we’re at it, we can support a multitude of export formats such that a software developer gets several outputs for the cost of Nixification.

5 Likes
20

That would be awesome, just created Export to .deb, .rpm… · Issue #93 · matthewbauer/nix-bundle · GitHub