Getting (Haskell) hpack data-files copied into derivation

Help
1

My Haskell web application has a number of runtime files (static content and other runtime config) that I need copying into the built derivation from the source package.

As my source package is hpack based, I’ve included a data-files: section at the top level, with appropriate paths under it. My understanding is that cabal2nix is supposed to be hpack-aware and I presumed therefore that files matching the glob patterns in a “data-files” list would be copied into the derivation, along with the main executable binary. However, I can’t seem to get this to work at all. After a nix-build I can always see my executable in the linked ‘result’ derivative, but that’s all it ever contains.

I’m sure there’s a nix-specific way to achieve this that I could use, instead of “data-files”, although I prefer the latter as, in theory, it would work for stack, cabal and nix builds.

2

I solved the original problem of the data files not apparently showing up in the result/binary derivation.

I learned that this is controlled by the
enableSeparateDataOutput = false;
… which I now have in my mkDerivation override.

One thing that I would like to control though is the prefix directory path that’s emitted before my files are added. For me this is: share/x86_64-osx-ghc-9.2.5/scoti2-0.0.6

This is a lot of prefix to cater for in all the places I want to reference the files from the original package. While I think I appreciate the fact that there may be different data files per architecture and package name, these discriminators are not pertinent in my case and just represent noise. So, is there any way to control this?

3

To answer your first question about where the data-files live, Nix has an idea of derivations with multiple outputs. This is used quite liberally all throughout Nixpkgs, so it is often a good idea to check for them if you feel something is missing in your derivation output. Here’s what it looks like in the repl:

$ nix repl '<nixpkgs>'
nix-repl> zlib
«derivation /nix/store/vc13znsdnkrbb8872gkjm7k259158vnb-zlib-1.2.13.drv»

nix-repl> zlib.outputs
[ "out" "dev" "static" ]

nix-repl> :b zlib
This derivation produced the following outputs:
  dev -> /nix/store/bsslcbcnrfcv6nl0jfha444nxjky7zxa-zlib-1.2.13-dev
  out -> /nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13
  static -> /nix/store/5zqngny5lssrqqfzh5ab9f809l99ms24-zlib-1.2.13-static

nix-repl> zlib.outPath
"/nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13"

nix-repl> zlib.out.outPath
"/nix/store/4xqki8nyafl26w2p71w988djvfmfz4x1-zlib-1.2.13"

nix-repl> zlib.dev.outPath
"/nix/store/bsslcbcnrfcv6nl0jfha444nxjky7zxa-zlib-1.2.13-dev"

nix-repl> zlib.static.outPath
"/nix/store/5zqngny5lssrqqfzh5ab9f809l99ms24-zlib-1.2.13-static"

So for zlib, depending on what you’re trying to do, you may be able to use zlib, or you may have to use zlib.dev or zlib.static. This is true for a lot of derivation in Nixpkgs.

The same thing applies to Haskell packages. For instance, the haskell package termonad makes use of multiple outputs:

nix-repl> haskellPackages.termonad.outputs
[ "out" "data" "doc" ]

The data files you’re expecting are in the .data output:

nix-repl> haskellPackages.termonad.data.outPath
"/nix/store/14hwjjnc7n9q5b7kln4lw9kzds33hzw9-termonad-4.2.0.1-data"

nix-repl> :b haskellPackages.termonad
This derivation produced the following outputs:
  data -> /nix/store/14hwjjnc7n9q5b7kln4lw9kzds33hzw9-termonad-4.2.0.1-data
  doc -> /nix/store/wb12n8gifnfjym89b1kq3f3scdiwgzww-termonad-4.2.0.1-doc
  out -> /nix/store/7k03zwkzhk7qlc7vrswwh7gbafwfxk2h-termonad-4.2.0.1

Then,

$ tree /nix/store/14hwjjnc7n9q5b7kln4lw9kzds33hzw9-termonad-4.2.0.1-data/
/nix/store/14hwjjnc7n9q5b7kln4lw9kzds33hzw9-termonad-4.2.0.1-data/
└── share
    └── ghc-9.0.2
        └── x86_64-linux-ghc-9.0.2
            └── termonad-4.2.0.1
                └── img
                    └── termonad-lambda.png

So instead of setting enableSeparateDataOutput = false, you could just use the haskell.packages.ghc925.your-package.data derivation to get your data files.

That’s a good question.

I imagine you’re aware that the data paths are available from your auto-generated _Paths module in your Haskell applications? So it should be no problem to get your assets from within your Haskell apps.

I’m guessing you want an easy way to reference your data assets from outside your Haskell application? I’m not sure what would be the best way to do that.

If I were to try to do it, I would add a postInstall (or something like this) handler to the Haskell package derivation and create a soft link within the .data derivation from a known path to the actual path of the asset files. So the resulting output might look like:

$ tree /nix/store/14hwjjnc7n9q5b7kln4lw9kzds33hzw9-termonad-4.2.0.1-data/
/nix/store/14hwjjnc7n9q5b7kln4lw9kzds33hzw9-termonad-4.2.0.1-data/
|----- my-assets   ->  ./share/ghc-9.0.2/x86_64-linux-ghc-9.0.2/termonad-4.2.0.1/
└── share
    └── ghc-9.0.2
        └── x86_64-linux-ghc-9.0.2
            └── termonad-4.2.0.1
                └── img
                    └── termonad-lambda.png

You can take a look at the generic Haskell builder to try to figure out what the data asset path will be:

I’m not sure off the top of my head what this would look like, but if you can’t figure it out, feel free to post again with what you’ve tried and I could take a look at it.

4

As always, thank you for such detailed replies.

For full context, this is actually what I’m trying to do.
I’m trying to configure the keter service in my configuration.nix, along these lines (which is really the pattern given by the author of the nix service for keter):

services.keter = {
    enable = true;
    keterPackage = pkgs.haskellPackages.keter;
    globalKeterConfig = {
      listeners = [{
        host = "*4";
        port = 3000;
      }];
    };
    bundle = {
      appName = "scoti2";
      domain = "mydomain.com";
      publicScript = ''
        export YESOD_PORT=$PORT
        export YESOD_STATIC_DIR=${scoti2Package}/share/static
      '';
      # TODO don't put the pass in the nix store ;)
      secretScript = ''
        export YESOD_PGDATABASE=<db>;
        export YESOD_PGPASS=<dbpass>;
        export YESOD_PGUSER=<dbuser>;
      '';  
      executable = pkgs.writeShellScript "run" ''
         ${scoti2Package}/bin/scoti2 ${scoti2Package}/share/config/settings.yml
      '';
    };
  };

As you can see, the config calls for two references into data flies that are in or associated with the package bundle. So, indeed these are outside of the yesod Haskell app per se. I could probably change my yesod app implementation to lookup paths from the Paths_scoti2.hs module as you suggest, given that these are defined as data-files in my package.yaml (hence cabal, hence cabal2nix’s derivation spec). However, the out-of-the-box yesod way to find configuration like this appears to be to get it from yaml config or specific environment variables, and the recipe for configuring services.keter is clearly trying to achieve both of these.

As it would be nice, to have short paths to my data files and ones that didn’t hardcode the specifics of the architecture and haskell compiler version and package version in these references for services.keter, I thought I’d try making a symbolic link in my mkDerivation’s postInstall block. However, I’m still struggling with that, as this involves getting access to all the symbols there to put in an ln -s command (I think!).

5

If I was trying to do something similar, I would probably just give a direct reference to the path of settings file, without going through the Haskell package’s data outputs:

    publicScript = ''
        export YESOD_PORT=$PORT
        export YESOD_STATIC_DIR=${./my/local/path/to/hask/pkg/static}
      '';

That way, you just don’t have to mess manually figuring out the static files path.

If, for some reason, you don’t want to reference the configuration files directly, and you do want to pass them in through Haskell data files, you could probably get a reference to the data files directory like the following. This is an example using Termonad, but you should be able to adopt it for your package:

let
  nixpkgs-src = builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/796b4a3c1d903c4b9270cd2548fe46f524eeb886.tar.gz";
    sha256 = "0163qw008i0liysq04g7ynvgc84mik1kvgz5qv6m690n4di0fw1q";
  };

  nixpkgs = import nixpkgs-src {};

in

with nixpkgs;

haskell.lib.overrideCabal haskellPackages.termonad (oldAttrs: {
  postInstall =
    let ghcName = "${haskellPackages.termonad.passthru.compiler.targetPrefix}${haskellPackages.termonad.passthru.compiler.haskellCompilerName}";
        sysPlusGhc = "${haskellPackages.termonad.system}-${ghcName}";
        termonadName = haskellPackages.termonad.name;
        fullDataPath = "share/${ghcName}/${sysPlusGhc}/${termonadName}";
    in
    oldAttrs.postInstall or "" + ''
      ln -sf "$data/${fullDataPath}" $data/data
    '';
})

Here’s what the .data output looks like:

$ tree /nix/store/lpchb3a9fbpgxk4dc3jl0g7z06hksncr-termonad-4.4.0.0-data
/nix/store/lpchb3a9fbpgxk4dc3jl0g7z06hksncr-termonad-4.4.0.0-data
├── data -> share/ghc-9.2.6/x86_64-linux-ghc-9.2.6/termonad-4.4.0.0
└── share
    └── ghc-9.2.6
        └── x86_64-linux-ghc-9.2.6
            └── termonad-4.4.0.0
                └── img
                    └── termonad-lambda.png
7

Thanks. I got your last example working great with one line modified to avoid a duplication of info:
fullDataPath = "share/${sysPlusGhc}/${termonadName}";

1 Like
8

…but (and I hope this is my last question!)…

I decided that I needed to copy these data files back out to the filesystem anyway (or at least some of them, as apparently yesod wants to create a static/tmp directory, which it can’t when the static data files are installed in the nix-store).

Consequently, I thought it would be sufficient to put something like this in my nixpkgs.nix postInstall phase:

cp -r "$out/${fullDataPath}/config" /var/www/config
cp -r "$out/${fullDataPath}/static" /var/www/static
cp -r "$out/${fullDataPath}/client" /var/www/client

I’m getting errors however, and in researching those, it seems that I didn’t understand that this code is called for all packages, and not just my final binary artifact (so $out can be for dependencies and for docs and other output).

I imagine there’s a way to test whether $out pertains only to the final package being built, so that I only attempt to copy these contents out when it’s the package where they actually exist.

9

In general, Nix performs builds in a sandbox, so you aren’t able to modify other parts of the file system.

If you want to copy files to /var, there are a few ways to do it that come to mind:

  1. Create a systemd unit that runs before the keter systemd service and creates the temp dir you need.
  2. I think NixOS has some sort of activationScript feature that allows you specify commands that should be run when running nixos-rebuild switch. You could use this to run your mkdir / cp commands.
  3. There may be some NixOS options to specify the creation of certain directories/files.

Although it may be easiest to just code the directory-creation/copying logic into your Haskell application. Or try to find if there is some way to specify to Yesod where it should create the temporary directory.

If you need more specific help, you might want to share all your Nix code so that people can see exactly what you’re trying to do.

10

Indeed, just ended up coding the directory copying in my app startup. Cheers.

1 Like