Derivation with two sources, split or not?

Hi all, I’m looking for advice on how to handle making a derivation.
This also happens to be the first derivation I’m making myself, so any other feedback to the structure is welcome! It’s a bit cobbled together from what I could find in the documentation and on Github :grin:

The application I’m trying to build is the Artemis. Compiling this app comes in two steps, the main repo (linked above) and the plugins repo.

I have a working derivation for just the program (no plugins). Which looks like this:

{ lib
, buildDotnetModule
, fetchFromGitHub
, dotnetCorePackages
, copyDesktopItems
, libX11
, libICE
, libSM
, fontconfig
, makeDesktopItem
}:

buildDotnetModule rec {
  pname = "artemis";
  version = "1.2024.0518.1"; # Latest versions here https://artemis-rgb.com/releases/linux

  src = fetchFromGitHub {
    owner = "Artemis-RGB";
    repo = "Artemis";
    rev = "7052ee38b6023a56dc385f277a52c65c6252afa2";
    sha256 = "sha256-ShgIAi0vp5q+5TPcLlJfEzfb3afo6atM45ZUFFgFeKU=";
  };

  projectFile = "src/Artemis.UI.Linux/Artemis.UI.Linux.csproj";
  nugetDeps = ./deps.nix;

  dotnet-sdk = dotnetCorePackages.sdk_8_0;
  dotnet-runtime = dotnetCorePackages.runtime_8_0;

  nativeBuildInputs = [ copyDesktopItems ];

  runtimeDeps = [
    libX11
    libICE
    libSM
    fontconfig # https://discourse.nixos.org/t/builddotnetmodule-runtimedeps-from-nugetdeps/23565
  ];

  postFixup = ''
    mv $out/bin/Artemis.UI.Linux $out/bin/artemis-rgb

    export ICON_DIR=$out/share/icons/hicolor/256x256/apps
    mkdir -p $ICON_DIR

    cp $out/lib/artemis/Icons/256x256/apps/artemis.png $ICON_DIR/Artemis.png
  '';

  desktopItems = [
    (makeDesktopItem {
      desktopName = "Artemis";
      name = "Artemis";
      exec = "artemis-rgb";
      icon = "Artemis";
      categories = [ "Utility" ];
      comment = meta.description;
    })
  ];

  meta = with lib; {
    homepage = "https://artemis-rgb.com/";
    description = "The unified RGB platform";
    license = licenses.free; # license is a PolyForm Noncommercial License 1.0.0
    mainProgram = "Artemis";
  };
}

From the logs I know where the built plugins should go:
[WRN] No built-in plugins found at /nix/store/f0j3igywpzxp5hck4bw7qpppq2yy6m1n-artemis-1.2024.0518.1/lib/artemis/Plugins, skipping CopyBuiltInPlugins

I’ve seen you can use multiple sources by using srcs = [...];
Currently the derivation looks like this. It does not link the plugins to the right directory yet, as I’m trying to figure out what the output is of the build of the plugins:

{ lib
, buildDotnetModule
, fetchFromGitHub
, dotnetCorePackages
, copyDesktopItems
, libX11
, libICE
, libSM
, fontconfig
, makeDesktopItem
}:

buildDotnetModule rec {
  pname = "artemis";
  version = "1.2024.0518.1"; # Latest versions here https://artemis-rgb.com/releases/linux

  sourceRoot = pname;

  srcs = [
    (fetchFromGitHub {
      owner = "Artemis-RGB";
      repo = "Artemis";
      name = pname;
      rev = "7052ee38b6023a56dc385f277a52c65c6252afa2";
      sha256 = "sha256-ShgIAi0vp5q+5TPcLlJfEzfb3afo6atM45ZUFFgFeKU=";
    })

    (fetchFromGitHub {
      owner = "Artemis-RGB";
      repo = "Artemis.Plugins";
      name = "artemis-plugins";
      rev = "d80cf160149fff7c9df9564fd06b8c87ecbfdfad";
      sha256 = "sha256-S5drT1W/7JNBKQd4Kh3onngAS/+a4sU88rtNJHND88k=";
    })
  ];

  projectFile = [
    "artemis/src/Artemis.UI.Linux/Artemis.UI.Linux.csproj"
    "artemis-plugins/src/Artemis.Plugins.sln"
  ];
  
  # https://github.com/NixOS/nixpkgs/blob/2e1ed6a2305c3c092265c758c1964ad0cc526ad8/doc/languages-frameworks/dotnet.section.md#generating-and-updating-nuget-dependencies-generating-and-updating-nuget-dependencies
  nugetDeps = ./deps.nix; 

  dotnet-sdk = dotnetCorePackages.sdk_8_0;
  dotnet-runtime = dotnetCorePackages.runtime_8_0;

  nativeBuildInputs = [ copyDesktopItems ];

  runtimeDeps = [
    libX11
    libICE
    libSM
    fontconfig # https://discourse.nixos.org/t/builddotnetmodule-runtimedeps-from-nugetdeps/23565
  ];

  # https://discourse.nixos.org/t/how-to-create-package-with-multiple-sources/9308
  preConfigure = ''
    mkdir artemis
    mv !(artemis) artemis

    chmod -R u+w ../artemis-plugins
    ln -s ../artemis-plugins .
  '';

  postFixup = ''
    mv $out/bin/Artemis.UI.Linux $out/bin/artemis-rgb

    export ICON_DIR=$out/share/icons/hicolor/256x256/apps
    mkdir -p $ICON_DIR

    cp $out/lib/artemis/Icons/256x256/apps/artemis.png $ICON_DIR/Artemis.png
  '';

  desktopItems = [
    (makeDesktopItem rec {
      desktopName = name;
      name = "Artemis";
      exec = "artemis-rgb";
      icon = "Artemis";
      type = "Application";
    })
  ];

  meta = with lib; {
    homepage = "https://artemis-rgb.com/";
    description = "The unified RGB platform";
    license = licenses.free; # license is a PolyForm Noncommercial License 1.0.0
    mainProgram = "artemis";
  };
}

The problem I’m running into is the deps.nix. I generated it by running the following in the Artemis.UI.Linux directory.

dotnet restore --packages out
nuget-to-nix out > deps.nix

But this tackles only one of the two projects that I want to compile


Long story short… I’ve tried manually combining the deps.nix of the two projects in projectFile attribute, but I’m still left with build errors. Now I’m considering splitting the two into separate derivations, and linking the plugins project with the projectReferences attribute somehow. But I’m not sure what the correct approach would be here.

Should I split the project into two, or keep them together?
And if I keep them together, how do I keep the deps.nix updateable and working?

Currently have the project split into two derivations

|- artemis-rgb
  |- default.nix
  |- deps.nix
|- artemis-rgb-plugins
  |- default.nix
  |- deps.nix

With artemis-rgb.nix looking like

{ lib
, buildDotnetModule
, fetchFromGitHub
, dotnetCorePackages
, copyDesktopItems
, libX11
, libICE
, libSM
, fontconfig
, makeDesktopItem
}:

buildDotnetModule rec {
  pname = "artemis";
  version = "1.2024.0518.1"; # Latest versions here https://artemis-rgb.com/releases/linux

  src = fetchFromGitHub {
    owner = "Artemis-RGB";
    repo = "Artemis";
    rev = "7052ee38b6023a56dc385f277a52c65c6252afa2";
    sha256 = "sha256-ShgIAi0vp5q+5TPcLlJfEzfb3afo6atM45ZUFFgFeKU=";
  };

  projectFile = "src/Artemis.UI.Linux/Artemis.UI.Linux.csproj";
  nugetDeps = ./deps.nix;
  packNupkg = true;

  dotnet-sdk = dotnetCorePackages.sdk_8_0;
  dotnet-runtime = dotnetCorePackages.runtime_8_0;

  nativeBuildInputs = [ copyDesktopItems ];

  runtimeDeps = [
    libX11
    libICE
    libSM
    fontconfig # https://discourse.nixos.org/t/builddotnetmodule-runtimedeps-from-nugetdeps/23565
  ];

  postFixup = ''
    mv $out/bin/Artemis.UI.Linux $out/bin/artemis-rgb

    export ICON_DIR=$out/share/icons/hicolor/256x256/apps
    mkdir -p $ICON_DIR

    cp $out/lib/artemis/Icons/256x256/apps/artemis.png $ICON_DIR/Artemis.png
  '';

  desktopItems = [
    (makeDesktopItem {
      desktopName = "Artemis";
      name = "Artemis";
      exec = "artemis-rgb";
      icon = "Artemis";
      categories = [ "Utility" ];
      comment = meta.description;
    })
  ];

  meta = with lib; {
    homepage = "https://artemis-rgb.com/";
    description = "The unified RGB platform";
    longdescription = ''
      Artemis is a lighting software for gamers that creates realistic lighting effects by using device location.
      It works across multiple brands and is open-source, allowing users to add their own devices, effects, and games.
      Artemis is designed to have minimal impact on gaming performance, ensuring a seamless experience.
    '';
    # license is a PolyForm Noncommercial License 1.0.0
    mainProgram = "Artemis";
  };
}

And artemis-rgb-plugins.nix looking like

{ lib
, buildDotnetModule
, fetchFromGitHub
, dotnetCorePackages
}:

buildDotnetModule {
  pname = "artemis-plugins";
  version = "1.2024.0518.1"; # What version to use for plugins repo?

  src = fetchFromGitHub {
    owner = "Artemis-RGB";
    repo = "Artemis.Plugins";
    rev = "d80cf160149fff7c9df9564fd06b8c87ecbfdfad";
    sha256 = "sha256-S5drT1W/7JNBKQd4Kh3onngAS/+a4sU88rtNJHND88k=";
  };

  projectFile = "src/Artemis.Plugins.sln";
  nugetDeps = ./deps.nix;

  dotnet-sdk = dotnetCorePackages.sdk_8_0;
  dotnet-runtime = dotnetCorePackages.runtime_8_0;
  # dotnetBuildFlags = [ "--property:UseArtemisNugetPackages=true" ]; # Maybe?

  meta = with lib; {
    homepage = "https://artemis-rgb.com/";
    description = "The unified RGB platform";
    longdescription = ''
      Artemis is a lighting software for gamers that creates realistic lighting effects by using device location.
      It works across multiple brands and is open-source, allowing users to add their own devices, effects, and games.
      Artemis is designed to have minimal impact on gaming performance, ensuring a seamless experience.
    '';
    # license is a PolyForm Noncommercial License 1.0.0
    mainProgram = "Artemis";
  };
}

However, I’m getting the following error when running the configurePhase

.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.unix.Microsoft.Win32.Primitives. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.unix.System.Console. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.any.System.Diagnostics.Tools. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.any.System.Diagnostics.Tracing. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.any.System.Globalization.Calendars. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.unix.System.IO.FileSystem. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.unix.System.Net.Primitives. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.unix.System.Net.Sockets. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.any.System.Text.Encoding.Extensions. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
.../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj : error NU1101: Unable to find package runtime.any.System.Threading.Timer. No packages exist with this id in source(s): nugetSource [.../artemis-rgb-plugins/source/src/Artemis.Plugins.sln]
  Failed to restore .../artemis-rgb-plugins/source/src/Other/Artemis.Plugins.Profiling/Artemis.Plugins.Profiling.csproj (in 204 ms).

I’m not sure what’s happening here, as the dependencies are listed in the deps.nix file, e.g.

(fetchNuGet { pname = "Microsoft.Win32.Primitives"; version = "4.3.0"; sha256 = "0j0c1wj4ndj21zsgivsc24whiya605603kxrbiw6wkfdync464wq"; })
(fetchNuGet { pname = "System.Console"; version = "4.3.0"; sha256 = "1flr7a9x920mr5cjsqmsy9wgnv3lvd0h1g521pdr1lkb2qycy7ay"; })
(fetchNuGet { pname = "System.Diagnostics.Tools"; version = "4.3.0"; sha256 = "0in3pic3s2ddyibi8cvgl102zmvp9r9mchh82ns9f0ms4basylw1"; })
...

Besides this error I’m seeing a potential issue with the order of the packages.
Going through the Directory.Build.Props file in the Artemis.Plugins repo, it looks like there is a reference to the locally compiled version of the Artemis repo.

<Reference Include="Artemis.Core">
    <HintPath>..\..\..\..\Artemis\src\Artemis.Core\bin\net8.0\Artemis.Core.dll</HintPath>
    <Private>false</Private>
</Reference>

Which does match with the build instructions in the project’s README

Want to build? Follow these instructions

  1. Create a central folder like C:\Repos
  2. Clone Artemis into <central folder>\Artemis
  3. Clone Artemis.Plugins master branch into <central folder>\Artemis.Plugins
  4. Open <central folder>\Artemis\src\Artemis.sln and build as Debug
  5. Open <central folder>\Artemis.Plugins\src\Artemis.Plugins.sln and build as Debug

But I’m not sure how I could translate this into a Nix derivation? :thinking:

I was originally going to suggest you use symlinkJoin, but your first question now is that you can’t compile plugins separately, is that true?

Yep, not looking positive on the buildPhase with 2390 errors :melting_face:

I’ve done a little ducttape fix by removing the pain project from the solution for the time by adding this to the artemis-rgb-plugins.nix derivation

postPatch = ''
  dotnet sln src/Artemis.Plugins.sln remove src/Other/Artemis.Plugins.Profiling/
'';

But that moves the errors to the buildPhase, where I get spammed with error messages like
error CS0234: The type or namespace name 'Core' does not exist in the namespace 'Artemis' (are you missing an assembly reference?) (repeat about 2390 times)

So compiling plugins separately is confirmed broken :sweat_smile: :+1:

So maybe the upstream wants us to put the plugins repository in a subdirectory of the main program repository? If that’s the case we just need to add some hooks.

Managed to get the plugins to compile locally by making use of the UseArtemisNugetPackages flag, and reverting back to commit cd66201, where the ArtemisRGB.UI.Shared package was last updated. Be sure to create a new deps.nix and the derivation compiles succesfully!

The specified flag was set by adding dotnetFlags = [ "-p:UseArtemisNugetPackages=true" ];

It looks like it dumps all the generated files in the same directory during the installPhase though, when they should all be in their own directory.

Also note this log statement:

warning NETSDK1194: The "--output" option isn't supported when building a solution. Specifying a solution-level output path results in all projects copying outputs to the same directory, which can lead to inconsistent builds. [/build/source/src/Artemis.Plugins.sln]
/nix/store/as1j1inzdjdaca6gdcfdr2bdsppgr3px-dotnet-sdk-8.0.300/sdk/8.0.300/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Publish.targets(223,5): error MSB3191: Unable to create directory "/nix/store/qlb90hzhishdp0pr3i4ahyx5dw9rxsyc-artemis-plugins-1.2024.219.2/lib/artemis-plugins/". Read-only file system : '/nix/store/qlb90hzhishdp0pr3i4ahyx5dw9rxsyc-artemis-plugins-1.2024.219.2' [.../artemis-rgb-plugins/source/src/Devices/Artemis.Plugins.Devices.Debug/Artemis.Plugins.Devices.Debug.csproj]

https://learn.microsoft.com/en-us/dotnet/core/compatibility/sdk/7.0/solution-level-output-no-longer-valid

The above source leads me to believe this could solve my issue, problem is I have no idea what the value should be for DIR, and how to get those variables in my derivation?
dotnetInstallFlags = ["-p:PackageOutputPath=DIR"];

I believe this is what’s being run that generates all those errors:

env dotnet publish ${project-} -p:ContinuousIntegrationBuild=true -p:Deterministic=true --output "${installPath-$out/lib/$pname}" --configuration "Release" --no-build ${runtimeIdFlags[@]} ${dotnetInstallFlags[@]} ${dotnetFlags[@]}

Thus, any way to remove the --output flag, and use the PackageOutputPath instead?