Creating a standardized audio production environment with flakes

I’m working on an open-source project that will make it easy for people to write music collaboratively. It creates an environment that works on all Linux distros that people can use for audio production, one which lots of people can run on their own machines.

The project currently uses an Arch Linux Docker image that has a variety of audio production packages installed. Still, we’ve hit a number of roadblocks when trying to integrate certain things, such as how to run X11 or Wayland apps in docker. Although we have worked around these things in the past, we keep finding new ones, and I’ve decided it might be better if we used Nix instead so we can avoid containers.

Currently, I have a shell script that runs nix-env -iA <list of audio packages> to get all the packages, and that seems to work most of the time, but when I have flakes enabled on my machine, it doesn’t work because it’s “incompatible with my profile.” What I would like to do is have a nix flake so we can control the versions of all the packages we’re using.

What we’re making is supposed to be used on systems where the user is the administrator, but ideally, it wouldn’t require administrative rights to run.


Below here describes a sizable problem I’ve run into with nix and some solutions I’ve tried to resolve it.

note: I am still new to Nix, and there are some concepts and bits of terminology below here that I’m still trying to grasp, so I apologize if I’m confusing. Just ask if you want me to explain something again.

One hurdle that using Docker doesn’t have to overcome is getting “audio plugins” to be discoverable. An audio plugin is a piece of software that runs within a plugin host, and a plugin host is a desktop application that can run audio plugins. For example, LSP plugins is a set of plugins available on Linux, and Ardour is a Digital Audio Workstation (DAW) that acts as a plugin host. The problem is because Nix likes to put things in unusual places, plugin hosts aren’t able to find the plugins. There are environment variables that tell plugin hosts where to look for plugins, but Nix doesn’t update them automatically. Here are all the environment variables, there is one for each type of plugin: LV2_PATH, VST3_PATH, VST_PATH, LXVST_PATH, LADSPA_PATH, and DSSI_PATH. For simplicity, I will talk about LV2 plugins below here.

Ideally, Nix would put all the plugins in a place that plugin hosts can see by default (probably in one of the standardized locations like ~/.lv2 or /usr/lib/lv2). When running the script I mentioned in the second paragraph on NixOS, it works fine because NixOS adds a path to the LV2_PATH environment variable to make it work: /etc/profiles/per-user/user/lib/lv2. The problem is the script doesn’t work if nix-command is enabled. I also tried nix-shell -p instead because that wouldn’t require admin access, but that doesn’t put the plugins in the right place. So, I have decided the best way to do this might be with a flake.


The main question I’m trying to get at is this:
How do I make a flake that installs packages from nixpkgs and puts the binaries in a place so other applications can find them while making it work on other Linux distros?

2 Likes

My answer may not cover all your questions, but just note that depending on the way you install softwares (system-wide, with nix-env…), the audio plugins will be installed in different places (which makes sense as a user without root access cannot modify /run/current-system), for instance here Calf package does not set `LV2_PATH` · Issue #45663 · NixOS/nixpkgs · GitHub people use this to configure the environment variable:

environment.variables = {
      DSSI_PATH   = "$HOME/.dssi:$HOME/.nix-profile/lib/dssi:/run/current-system/sw/lib/dssi";
      LADSPA_PATH = "$HOME/.ladspa:$HOME/.nix-profile/lib/ladspa:/run/current-system/sw/lib/ladspa";
      LV2_PATH    = "$HOME/.lv2:$HOME/.nix-profile/lib/lv2:/run/current-system/sw/lib/lv2";
      LXVST_PATH  = "$HOME/.lxvst:$HOME/.nix-profile/lib/lxvst:/run/current-system/sw/lib/lxvst";
      VST_PATH    = "$HOME/.vst:$HOME/.nix-profile/lib/vst:/run/current-system/sw/lib/vst";
    };

So a solution it to configure the environment variables like above (you can even add a specific package to this list in case the above list of path is not exhaustive by prepending ${yourprogram}/lib/lv2). You can configure the variables in many different ways, but a first solution is to use a wrapper script (see makeWrapper) to set this before starting your software, or use home-manager to configure it more easily on the whole user session (note that home manager also provides cool functionalities to run systemd user “daemons” that may be practical to start pipewire or alike). You can also write a file and ask the user to source it in bashrc…

1 Like

Thanks for the tip! It sounds like writing this in such a way that home-manager can take care of as much as possible would be really advantageous. How would I use home-manager in a flake? Below is some basic code I have for testing. It’s mostly cobbled together from various places on the internet, so I don’t understand why some parts exist, but I do understand what 90% of it is doing.

What I do to test that it works is I run nix develop, then I run ardour6, open a blank project, go to add a track by right-clicking in the blank space on the left, select midi track on the left, then click the plugins list to see if Helm is there. If it’s there, everything works™. If it isn’t there, it means the plugins won’t be seen by Ardour.

# flake.nix
{
  description = "Audio production environment";

  inputs = {
    nixpkgs = { url = "github:NixOS/nixpkgs/nixpkgs-unstable"; };
    flake-utils = { url = "github:numtide/flake-utils"; };
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        inherit (nixpkgs.lib) optional;
        pkgs = import nixpkgs { inherit system; };

      in
      {
        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            ardour
            helm
          ];
        };
      }
    );
}

How would I write this code to take advantage of home-manager?

About the environment variables, is home-manager able to update the plugin path environment variables for me? Also, I’ve noticed the plugins are all put into /nix/store/hash-program-version. When you say I could ask the user to source a file in their bashrc, are you suggesting that I use that to add each of these paths in the store to the plugin path variables?

So here I demonstrate how plugins can be added to a single program only. You don’t even need to install the plugins in your system or use home manager, you can simply run:

nix run 'git+https://gist.github.com/50988e8b091e169b67eb082f1c77e533.git?ref=main#ardour' --refresh

and it will run ardour with the helm plugin for you:

image

Note that the url is just pointing to the second revision of a gist I created here and the refresh is just used to be sure you always run the latest version.

For reference, here is the full flake (no additional file is needed):

{
  description = "Audio production environment";

  inputs = {
    nixpkgs = { url = "github:NixOS/nixpkgs/nixpkgs-unstable"; };
    flake-utils = { url = "github:numtide/flake-utils"; };
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
        lib = pkgs.lib;
        # Plugins to load (lv2/…)
        myPlugins = [ pkgs.helm ];
        # lv2_path will look like:
        # ${helm}/lib/lv2
        # this is just more general to loop over all plugins as makeSearchPath adds lib/lv2 to all
        # outputs and concat them with a colon in between as explained in:
        # https://nixos.org/manual/nixpkgs/stable/#function-library-lib.strings.makeSearchPath
        # TODO READER: you should do the same for all other plugin formats, this is left as an
        #              exercice for the reader ;-)
        lv2_path = (lib.makeSearchPath "lib/lv2" myPlugins);
        wrapMyProgram = { programToWrap, filesToWrap ? "*" }: pkgs.runCommand
          # name of the program, like ardour-with-my-plugins-6.9
          (programToWrap.pname + "-with-my-plugins-" + programToWrap.version)
          {
            nativeBuildInputs = with pkgs; [
              makeBinaryWrapper
            ];
            buildInputs = [
              programToWrap
            ];
          }
          ''
            mkdir -p $out/bin
            for file in ${programToWrap}/bin/${filesToWrap};
            do
              filename="$(basename -- $file)"
              # TODO READER: should do the same for all plugins formats (you can have multiple prefix arguments)
              # to see the content of the wrapper to debug, use less as it is in binary format (for compatibility
              # with MacOs)
              makeWrapper "$file" "$out/bin/$filename" \
                --prefix LV2_PATH : "${lv2_path}";
            done
          '';
      in
      rec {
        # Executed by `nix build .#<name>`
        packages = flake-utils.lib.flattenTree {
          # This creates an entry self.packages.${system}.ardour with the wrapped program.
          ardour = wrapMyProgram { programToWrap = pkgs.ardour; };
        };
        # Executed by `nix run .#<name>`
        apps = {
          ardour = {
            type = "app";
            program = "${self.packages.${system}.ardour}/bin/ardour6";
          };
        };
        # Used by `nix develop` (not really needed if you use nix run)
        devShell = pkgs.mkShell {
          buildInputs = with pkgs; [
            self.packages.${system}.ardour
          ];
        };
      }
    );
}

For now it is using only lv2 plugins but other types of plugins can be trivially added (this is left as an exercice, note that wrapProgram can accept multiples prefix arguments).

The code itself is not really complicated (maybe just unusual if you are not used to nix). I put the list of plugins in a variable myPlugins and from this list I generate a variable lv2_path that will basically be of the form:

${plugin1}/lib/lv2:${plugin2}/lib/lv2…

Then I create a function wrapMyProgram (it will be handy to wrap easily many programs) that takes as argument the program to wrap (ardour here) and outputs a derivation that morally sets LV2_PATH=${lv2_path}:$LV2_PATH, i.e.:

LV2_PATH=${plugin1}/lib/lv2:${plugin2}/lib/lv2:$LV2_PATH

before running the program (by default it does that for all binaries in the bin folder of your program). The rest is just flake boilerplate to properly add ardour in the list of programs that can be run.

Warning: I have no time to test this part, let me know if it does not work as expected:

If you want to create a flake module for home-manager or nixos (mostly useful to configure services I would guess), first check here how you user should configure home manager with flake (or the nixos doc for nixos). Then, add the flake configuration you want in your flake, for instance right after the devShell part, say under homeModules.myAudioSetup (or nixosModules.myAudioSetup for nixos), as far as I understand the exact name does not really matters as soon as you refer it properly later and does not take a name that conflicts with the other conventions. Then in the user’s flake you should be able to add the audio flake as an input:

  inputs.yourAudioFlake.url = "github:your/flake";

adds it in the argument of output:

  outputs = { self, nixpkgs, flake-utils, yourAudioFlake }: {

and in the module do something like:

        modules = [
          ./home.nix
         yourAudioFlake.homeModules.myAudioSetup
        ];

At least it is what you should do for nixos modules as documented here and I guess the same should also work for home manager modules (but again I have never tested).

Hope it helps! Let me know if you have further questions.

3 Likes

Thank you for the help, @tobiasBora! I have taken what you have given me and made this codeberg.org/PowerUser/universal-studio! Are there any ways you think this could be improved? Also, is there a way to make the nix run command in the readme any shorter? Right now, it has some really long --extra-experimental-features flags in it.

Again, thank you so much for your help!

That sounds great, congrats :wink: (And I may actually use it, at least to get the name of the great plugins)

Regarding your “long” command, until flake is not experimental you don’t have much choice, you need to enable it somehow. That said, you can enable it once for all, and then just use nix run … without the flags.

Regarding improvements, I can imagine a few improvements:

Package yourself the plugins that are not yet in nixpkgs (and eventually upload them upstream)

I see that you mention some plugins that are not yet in nixpkgs… That’s not so much of an issue, you can certainly package it yourself easily! I created a tutorial here if you want to get started with packaging. As far as I see, it should be quite easy to build wolf-spectrum and dpf-plugins, you just need to fetch the repository with submodules = true; for wolf-spectrum as the git repo has submodules, and certainly enable cmake for it as well, and configure the appropriate (c)makeFlags as explained in the doc (dpf may be even easier as it is a simple Makefile).

In case it can help, I packaged at some point Ildaeil to learn how to package stuff and it was looking like:

{ lib
, stdenv
, fetchFromGitHub
, cmake
, pkg-config
# Not sure which one I should keep (at least libGL for sure)
, liblo
, freetype, libX11, libXrandr, libXinerama, libXext, libXcursor, libGL
, gtk2, gtk3
, libjack2, alsa-lib, alsa-tools
}:

stdenv.mkDerivation rec {
  pname = "Ildaeil";
  version = "unstable-6b303b3059c";

  src = fetchFromGitHub {
    owner = "DISTRHO";
    repo = pname;
    rev = "6b303b3059c";
    sha256 = "sha256-wqhArYjPA0yVedW654nCFQTwKi02fvImHkW4W3xoBfc=";
    fetchSubmodules = true;
  };

  nativeBuildInputs = [
    # cmake
    pkg-config
  ];
  buildInputs = [
    liblo
    freetype
    libX11 libXrandr libXinerama libXext libXcursor libGL
    gtk2 gtk3
    libjack2 alsa-lib alsa-tools
  ];

  makeFlags = [ "PREFIX=$(out)" ];
  
  patchPhase = ''
    patchShebangs dpf/utils/generate-ttl.sh
  '';
}

Refactor to avoid manually repeating the same code for all packages

I guess that you may not want to repeat yourself many times for all the applications that you want to wrap (as I can see from your comment). A solution is to use map (that applies a function to a list, so map (x: x+10) [1,2,3] = [11,12,13]) and listToAttrs (that turn a list [ {name = x1, value = y1} {name = x2, value = y2} ]) into an attrset {x1 = y1; x2 = y2}).

To give an example (that must of course be adapted to your case):

$ nix repl
nix-repl> :p builtins.listToAttrs (builtins.map (app: {name = app; value = {type = "app"; program = "wrapped version ${app}";};}) myApps)
{ app1 = { program = "wrapped version app1"; type = "app"; }; app2 = { program = "wrapped version app2"; type = "app"; }; }

You just need to adapt it to your case :slight_smile:

1 Like

Hello, @tobiasBora!

I’ve taken some time to write some other parts of the project. There is now a shell script called universal-studio.sh that can list and launch any of the programs from the flake.nix. What’s cool is it should work anywhere because it can use nix-portable if it doesn’t find nix.


I’m coming back here, however, because I’m a little confused about how I should use listToAttrs and map like you were suggesting. Is there a place you would recommend I go to learn things like this? The nix programming language is still a bit foreign to me. (Especially the syntax, it’s hard to tell what’s what and why some things happen.)

Another thing is I’ve noticed that it may not be possible to use something like listToAttrs everywhere because, for example, the binary for each program doesn’t always have the same as the package name; like how ardour provides a binary called ardour6. Is there a way to automatically pick the binary? It might also not be worth doing if it would be too complicated.


Something else is I’ve encountered an issue where VST and VST3 plugins are no longer being located by programs like Ardour. I checked the plugin environment variables (VST3_PATH, etc.), and I didn’t see any plugins added to the folders that weren’t already there. Do know what might be causing this?

If you have time and want to debug it, here’s a oneliner that might help you - it lists the contents of all the plugin environment variable directories:

eval 'printf "%s\\n" $'{LV2,VST,VST3,LXVST,DSSI,LADSPA}'_PATH | grep -o "[^:]\+";' | xargs ls


Thanks for all your help!

That looks great! So let’s see each issue at a time:

Nix language

So to learn about the nix language you always have the nix manual and the Nix Pills that look more like a tutorial and are certainly easier to start with for the first two pages (you don’t need stuff regarding derivations).

You can also try to play with nix using the interpreter:

$ nix repl '<nixpkgs>'
nix-repl> 1+1
2

nix-repl> map (x: 2*x) [1 2 3]
[ 2 4 6 ]

nix-repl> pkgs.ripgrep.meta.mainProgram
"rg"

Importantly, Nix is a functional language, which is (roughly speaking) a programming paradigm in which functions play a central role, and in which you don’t have mutable variable (so variables can’t be changed like in imperative programming) or loops. Instead of loops, you use recursive functions (functions that call themselves)… or more often builtins functions like map or fold that implement the recursion for you (you really rarely need to code recursive functions in nix, most of the time use builtins functions that are available here, see also stuff in lib for more advanced stuff, all functions from builtins/lib are listed here as well). It has multiple advantages as it’s easier to automatically optimize code for parallelization (the compilator has more freedom to optimize your code, see eg Haskell), it also allows you to code stuff in a more concise way, you can use laziness to only evaluate an expression when you need it (this way infinite lists does make sense) and more…

To give maybe two examples of the most used constructions, the nix code:

map (x: 2*x) [1 2 3]

(that applies a function 2x to a list and returns [2 4 6]) is an equivalent of the python imperative code:

l = [1, 2, 3]
l2 = []
for x in l:
    l2 += [x*2]

(note that python also allows some sort of functional programming, so in python you would use list comprehensions coming from Haskell like [2*x for x in l ] to do the above map)

Similarly you can fake loops using:

foldl f acc list

which is doing the equivalent of the python code:

for x in list:
     acc = f acc x
return acc

(acc is called the accumulator as it accumulates the result of the previous loops) i.e:

foldl f acc [x_1 x_2 ... x_n] == f (... (f (f acc x_1) x_2) ... x_n)

To give a more concrete example, you could sum the elements of the lists using

nix-repl> lib.lists.fold (a: b: a+b) 0 [1 2 3]
6

as internally it will do ((0 + 1) + 2) + 3) = 6 since a: b: a+b is the function that takes two arguments a and b and returns a+b.

Similarly you could join multiple attribute sets using:

nix-repl> :p lib.foldl lib.recursiveUpdate {} [{a.b = 42;} {a.c = 44;}]
{ a = { b = 42; c = 44; }; }

(the :p is just used in nix repl to display the result)
since lib.recursiveUpdate A B merges the two sets A and B.

Functional programming is not really ‘complex’ (everybody knows functions, right?), but it can be a bit disturbing to use when you are not used to it. You can find plenty of tutorials on functional programming online, see e.g. So You Want to be a Functional Programmer (Part 1) | by Charles Scalfani | Medium. In nix (and in many functional languages) I guess the most disturbing thing is that functions don’t have return since it returns by default the last value, and the variable definitions are like let x = 5; in … where ist a code block that can refer to x.

Usage of listToAttr

So your goal is to rewrite this whole section into a code that automatically generate this same attribute set given [pkgs.ardour pkgs.audacity pkgs.bespokesynth …]. Actually I’m thinking now that listToAttrs is not even the best tool to use, you can first define next to your other definitions a function myCreatePackagesAppsShell that creates the attribute for a single package (only partially tested):

nix-repl> myCreatePackagesAppsShell = myPackage: {                                                                      
            packages = flake-utils.lib.flattenTree { "${myPackage.pname}" = wrapProgram { programToWrap = myPackage; }; };
            apps = { "${myPackage.pname}" = { type = "app"; program = getExecutablePath self.packages.${system}.${myPackage.pname}; }; }; # See below for definition of getExecutablePath
          }                        

nix-repl> myCreatePackagesAppsShell pkgs.audacity                  
{ apps = { ... }; packages = { ... }; }

Then if you compute map myCreatePackagesAppsShell [ pkgs.audacity pkgs.ardour ] you will get a list of attributes as described the line above: what you actually want is to merge all such attributes using something like that (put that directly in place of this whole code):

lib.lists.foldr lib.recursiveUpdate {} (map myCreatePackagesAppsShell myApps)

(see the above explaination of fold/foldr/foldl to understand why it merges the sets, and you can find more ressources online like here)

You may have noticed that we did not considered yet the devShell but it should be easy to add it as well using:

lib.lists.foldr lib.recursiveUpdate {} [
  # Attribute set for the packages/app stuff 
  (map myCreatePackagesAppsShell myApps))
  # Shell
  { devShell = pkgs.mkShell {
      buildInputs = map (myPackage: self.packages.${system}.${myPackage.pname}) myApps ;
    };
  }
]

(not tested)

Name of programs: meta.mainProgram (+overrideAttrs if needed)

Indeed, it can be an issue that programs don’t always have an executable name whose name is the name of the program. nix run also has this issue and people introduced here a new way to deal with that case: programs that have a binary different from their own name should add an attribute meta.mainProgram with the name of the binary. It is for instance the case of ripgrep or bespokesynth:

nix-repl> pkgs.bespokesynth.meta.mainProgram
"BespokeSynth"

Unfortunately not all programs follow this convention yet (looking at you ardour… maybe you can do your first contribution to nixpkgs to add that to ardour following the code in ripgrep!) If you still want to do that in your project without waiting for nixpkgs to accept your pull request, you can also override it locally:

let # in same let block as myApp…
  myArdour = pkgs.ardour.overrideAttrs (finalAttrs: previousAttrs: { meta.mainProgram = "ardour6";});
  # Apps = …
in # replace pkgs.ardour with myArdour

(note that it should not produce any recompilation, you can also inject it in an overlay, but it’s not even required, you can define it right before Apps and just refer to myArdour indeed of pkgs.ardour later in your code)

You can then access this name using:

nix-repl> myArdour.meta.mainProgram
"ardour6"

Since many programs still use the pname as the name of the binary (like audacity), you can create your function that picks meta.mainProgram if it exists and fallback to pname otherwise:

nix-repl> getExecutablePath = program: "${program}/bin/" + (program.meta.mainProgram or program.pname)  

nix-repl> getExecutablePath pkgs.audacity                                                                
"/nix/store/q8vzngi38d1mgr8p22zkscff0pb29b1g-audacity-3.1.3/bin/audacity"

nix-repl> getExecutablePath myArdour                                                                     
"/nix/store/r0my44qdja5mrfg08ihnj840cazr88y6-ardour-6.9/bin/ardour6"

Issues with path

I’m not sure to understand your issue: which ardour are you talking about: the one from nix run .#ardour (which would be strange since you wrap it correctly as far as I see), or the one from the system (which is completely normal since you only locally change the environment variable in your wrapper, which don’t propagate to the rest of the system unless you take further actions for that)? Also, your oneliner has no chance to give any folder in /nix/store for the reason mentioned above: the path are only defined in the wrapper of your program.

PS: nix --extra-experimental-features flakes --extra-experimental-features nix-command can be shorten into nix --extra-experimental-features 'flakes nix-command'.
PPS: regarding this comment you can either just replace the flake with . this way it refers to the flake in the current folder (since you instruct you users to download the repository) or you can use git+https://codeberg.org/PowerUser/universal-studio.git if you prefer them to always get the latest version (you may want to add --refresh in the nix run command in case it does not download the latest version automatically).

2 Likes

Thanks for the resources! I fixed the problem with ardour not seeing the plugins, but I’m not sure how. It just sorta decided to work after I read your message here.

I’ve slowed the project down a bit lately so I can get more comfortable with the nix language by writing my system configuration in a flake (of course), as well as working on some packages to submit to nixpkgs. Seeing the examples you brought up here really helped me to see nix as an actual programming language rather than something like JSON, which is how I saw it before.

It might seem like things are moving a bit slowly right now, but it’s just because I’m learning to use the tools. Perhaps making an ambitious project with nix within a week of learning about it wasn’t the smartest thing to do :stuck_out_tongue: (…or maybe it was :​)

Just thought I’d post a bit of an update here because I’ve been quiet for a while.

PS: I’m in the NixOS matrix room now! I’m @poweruser64:matrix.org, feel free to say hi if you see me around :slight_smile:

Ahah thanks for your feedback no worries, do whatever you like, no pressure ^^ Welcome to the great Nix world ! :smiley: