Propagate output with overlay between two linked flakes?

HI,

I’m beginner in the world of flakes, i found that my problem is simple on paper, but really hard to resolve using current documentation into nix. I found part of solution alone, and part with the help of a nix redditer : https://stackoverflow.com/questions/69873273/need-to-build-specific-version-of-pandoc-into-poetry2nix-nixos-flake/69987937#69987937

I have two flakes A & B.

  • Flake A build a Python package using poetry2nix : myPythonApp
  • Flake B build a Haskell lib using haskell.nix : a specific pandoc version
  • Flake A need Flake B to run correctly using poetry : poetry run /results/bin/myPythonApp

The A flake is defined like that

{
  description = "Application packaged using poetry2nix";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs";
  inputs.poetry.url = "github:nix-community/poetry2nix";

  outputs = inputs@{ self, nixpkgs, flake-utils, poetry }:
    let
      overlay = final: prev: ({
        # FIXME: Change to some more verbose name
        myPythonApp = final.poetry2nix.mkPoetryApplication {
          projectDir = ./.;
          python = prev.python39;
          propagatedBuildInputs = [
            prev.graphviz
            prev.bash
            prev.wget
            prev.findutils
          ];
        };
      } //
      # This is kind of ugly, but required as it is not possible to return array in flake
      # It merges poetry overlay with this overlay
      (poetry.overlay final prev));
    in
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; overlays = [ overlay ]; };
      in
      {
        defaultPackage = pkgs.myPythonApp;
      }
    ) // {
      # Provide overlay outside of flake-utils so that it will not have
      # system prefix.
      inherit overlay;
    };
}

Running nix build create a binary python package /results/bin/myPythonApp

{
  description = "PandocProject";
  inputs.haskellNix.url = "github:input-output-hk/haskell.nix";
  inputs.nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.myPythonApp.url = "path:/home/mateusz/so/flakeA"; # FIXME: Change to your url
  outputs = { self, nixpkgs, flake-utils, haskellNix, myPythonApp }:
    flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
      let
        overlays = [
          myPythonApp.overlay
          haskellNix.overlay
          (final: prev: {
            # This overlay adds our project to pkgs
            pandocProject = final.haskell-nix.cabalProject {
              cabalProjectFreeze = null;
              cabalProject = null;
              cabalProjectLocal = null;

              # In the `final` attribute there is your python script !
              # Not sure how you want to use it
              # It is avaible as `final.myPythonApp?` 
              
              compiler-nix-name = "ghc8104";
              name = "pandocProject";

              src = final.fetchFromGitHub {
                name = "pandoc";
                owner = "jgm";
                repo = "pandoc";
                rev = "d05460d00d7c9af3b4913f1760ea385a7d855d84";
                sha256 = "1a3kwag6j13b42zhzxiwlzabsc6c9jppiwv9j8gbnf2k1yb84kdm";
              };

              pkg-def-extras = with final.haskell.lib; [ ];
            };
          })
        ];
        pkgs = import nixpkgs { inherit system overlays; };
        flake = pkgs.pandocProject.flake { };
      in
      flake // {
        # Built by `nix build .`
        defaultPackage = flake.packages."pandoc:exe:pandoc";
      });
}

Building B flake produce a pandoc binary in /results/bin/pandoc

My Problem :

With overlays, i merge A and B into B, ok, but now, i don’t know how i do to build a /results/bin folder or a shell/develop context that contain BOTH the A package (mypythonapp) AND the B package (pandoc)

Thx …

1 Like

I usually have this in my flakes:

overlays = [ overlayA overlayB ];
overlay = nixpkgs.lib.composeManyExtensions overlays;

Downstream flakes just need to consume the overlay I expose, and they should get the other overlays that I consume.

To have the shell expose both, you just want to use final.callPackages (or just pull from the final package set).

1 Like

Thanks @jonringer

I try to change my flakes based on your answer, but, how do you get final var in mkShell ?

{
  description = "PandocProject";
  inputs.haskellNix.url = "github:input-output-hk/haskell.nix";
  inputs.nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.wiki-python.url = "git+https://gitlab.huma-num.fr/gt-notebook/wiki.git?ref=main";

  outputs = { self, nixpkgs, flake-utils, haskellNix, wiki-python }:
    {
      overlay = nixpkgs.lib.composeManyExtensions  [
      wiki-python.overlay
      haskellNix.overlay
        (final: prev: {
          # This overlay adds our project to pkgs
          pandocProject = final.haskell-nix.cabalProject {
              cabalProjectFreeze = null;
              cabalProject = null;
              cabalProjectLocal = null;

              compiler-nix-name = "ghc8104";
              name = "pandocProject";

              src = prev.fetchFromGitHub {
                 name = "pandoc";
                 owner = "jgm";
                 repo = "pandoc";
                 rev = "d05460d00d7c9af3b4913f1760ea385a7d855d84";
                 sha256 = "1a3kwag6j13b42zhzxiwlzabsc6c9jppiwv9j8gbnf2k1yb84kdm";
              };
              pkg-def-extras = with final.haskell.lib; [];
            };
        })
      ];
    } // (flake-utils.lib.eachSystem ["x86_64-linux"] (system:
      let
        pkgs = import nixpkgs {
            inherit system;
            overlays = [self.overlay];
        };
        flake = pkgs.pandocProject.flake {};
      in {
        defaultPackage = flake.packages."pandoc:exe:pandoc";

        devShell = pkgs.mkShell {
            buildinputs = with pkgs; [
            #calling final ???
            ];
        };
    }
    ));

I have this when i run flake show, i’m not sure overlay are correctly consumed.

 
git+file:///home/xxx/pandoc2nix
├───defaultPackage
trace: No index state specified for pandocProject, using the latest index state that we know about │   └───x86_64-linux: package 'pandoc-exe-pandoc-2.14.2'
├───devShell
│   └───x86_64-linux: development environment 'nix-shell'
└───overlay: Nixpkgs overlay

Since you already applied the overlay here:

        pkgs = import nixpkgs {
            inherit system;
            overlays = [self.overlay];
        };

Using packages from the package set after application of the overlays should reflect the changes in those overlays.

In other words, that looks about right to me.

Ok, strange, with this new configuration nix build do nothing so i probably miss something in this new definition …

Edit : I miss the recursion, i correct the flakes with :

      let
        pkgs = import nixpkgs {
            inherit system;
            overlays = [self.overlay];
        };
        flake = pkgs.pandocProject.flake {};
      in flake // {
        defaultPackage = flake.packages."pandoc:exe:pandoc";

        devShell = pkgs.mkShell {
            buildinputs = with pkgs; [
            #calling final ???
            ];
        };
    }
    ));

And now nix build return an infinite recursion, :

error: infinite recursion encountered

       at /nix/store/lvbp2s8my5wvqwk0pagxf5xb0ixrg886-source/flake.nix:39:25:

           38|             inherit system;
           39|             overlays = [self.overlay];
             |                         ^
           40|         };

I would define the overlays in a let block, then reference them from the let scope. You’re adding the overlays to your flake, then referencing them through self.overlay.

Example: https://github.com/jonringer/nix-template/blob/2e7238422489a608453704e9f5b897b2b1c0c2e7/flake.nix

Ok,

I try to change the structure of the flake based on your example :


  outputs = { self, nixpkgs, flake-utils, haskellNix, wiki-python }:
      let
      my-overlay = nixpkgs.lib.composeManyExtensions  [
      wiki-python.overlay
      haskellNix.overlay
        (final: prev: {
          pandocProject = final.haskell-nix.cabalProject {
              cabalProjectFreeze = null;
              cabalProject = null;
              cabalProjectLocal = null;

              compiler-nix-name = "ghc8104";
              name = "pandocProject";

              src = prev.fetchFromGitHub {
                 name = "pandoc";
                 owner = "jgm";
                 repo = "pandoc";
                 rev = "d05460d00d7c9af3b4913f1760ea385a7d855d84";
                 sha256 = "1a3kwag6j13b42zhzxiwlzabsc6c9jppiwv9j8gbnf2k1yb84kdm";
              };
              pkg-def-extras = with final.haskell.lib; [];
            };
        })

      ];

      pkgs = system: import nixpkgs {
          overlays = [my-overlay];
          inherit system;
      };

      in (flake-utils.lib.eachSystem ["x86_64-linux"] (system: rec {

        packages = pkgs system;

      #flake = pkgs.pandocProject.flake {};
      defaultPackage = packages.pandocProject."pandoc:exe:pandoc";
    }));
    }

I understand the first part and the principle of recursive and overlay, which compile… BUT i don’t understand how to modify the final part that build package .

      flake = pkgs.pandocProject.flake {};
      defaultPackage = packages.pandocProject."pandoc:exe:pandoc";

return error: attribute 'pandoc:exe:pandoc' missing

I’m going to be mad with nixos syntax :scream: I try so many different syntax / things to do the same thing for such a simple use case …

To be sure, based on your example, i need to create an overlay.nix that contain two file ? :

  • one file with my composed overlay nixpkgs.lib.composeManyExtensions [ ... ]
  • one file with mkShell
  • then flatten
  • then call apps.nix-template = utils.lib.mkApp { drv = packages.nix-template; };

?

for debugging flakes, I usually do:

$ nix repl
nix-repl> :lf .
loaded 6 variables.

nix-repl> packages.<tab><tab>

and poke around.

1 Like

Thanks,
Yeah, i suppose this is a bad news …

nix-repl> packages             
{ x86_64-linux = { ... }; }

nix-repl> packages.x86_64-linux
{ }

Edit :

Ok i finally found a way to get one of the two composed derivation using flatten :

legacyPackages = pkgs system;

    packages = flake-utils.lib.flattenTree {
        inherit (legacyPackages) pandocProject wikiApp;
    };

    defaultPackage = packages.wikiApp;

This is better, i’m near the end ! but i don’t know how to export the two packages using defaultPackages now, any idea @jonringer ?

defaultPackages is just a QoL feature, I wouldn’t concern yourself too much about it. If you do have two equally important packages, then maybe having neither as the default is the correct choice.

1 Like

@jonringer Yes, but if i don’t define defaultPackages nix build failed …

error: flake 'git+file:///home/XXX/TRASH/pandoc2nix' does not provide attribute 'packages.x86_64-linux.defaultPackage.x86_64-linux', 'legacyPackages.x86_64-linux.defaultPackage.x86_64-linux' or 'defaultPackage.x86_64-linux'

To define two equally package, like you say, i need to use this for example ?

flake-utils.lib.mkApp {} ?

If you don’t have a default, you need to specify what to build, eg. nix build .#foo

1 Like

Thanks, so, i have a problem in my drv definition, i don’t know why but only wikiApp drv is found, pandocProject is masked or not well defined …

nix build .#wikiApp 
ok

nix build .#pandocProject
error: flake output attribute 'legacyPackages.x86_64-linux.pandocProject' is not a derivation

In nix repl :

nix-repl> packages.x86_64-linux        
{ wikiApp = «derivation /nix/store/4s3aki4cldddgi83r7dp1lbdjanb3k7l-python3.9-notebook-wiki-0.0.1.drv»; }

It seems haskellnix use the flake= attribute (see https://input-output-hk.github.io/haskell.nix/tutorials/getting-started-flakes/#scaffolding ) in this order ;

let
      overlays = [ haskellNix.overlay [...]
      pkgs = import nixpkgs { inherit system overlays; inherit (haskellNix) config; };
      flake = pkgs.helloProject.flake {...}
in flake // { 
...
}

What this default snippet of code do ?

Ok, using this, each call nix build .#wikiApp or .#pandoc one create a package, that works, now i need to merge both !

        apps.wiki = flake-utils.lib.mkApp { drv = packages.wikiApp; };
        apps.pandoc = flake-utils.lib.mkApp { drv = packages.pandocProject; };

At @jonringer , i try to reproduce the structure of flakes with overlay you give me previously :

outputs = { self, nixpkgs, flake-utils, haskellNix, wiki-python }:
      let
      my-overlay = import ./overlay.nix {inherit haskellNix wiki-python;};      
      pkgs = system: import nixpkgs {
          overlays = [my-overlay ];
          inherit system;
      };

      in (flake-utils.lib.eachSystem ["x86_64-linux"] (system: rec {
        legacyPackages = pkgs system;
        packages = flake-utils.lib.flattenTree {
            inherit (legacyPackages)
            wikiPandoc;
        };

        apps.wikiPandoc = flake-utils.lib.mkApp { drv = packages.wikiPandoc; };

        devShell = legacyPackages.mkShell {
          packages = [
            packages.wikiPandoc
          ];
        };
    }));
    }

overlay.nix

{nixpkgs, haskellNix, wiki-python}:
final: prev: {
  wikiPandoc = nixpkgs.lib.composeManyExtensions  [ {
      pandoc = prev.callPackage ./pandoc {inherit haskellNix;};
      #wiki = prev.callPackage wiki-python.overlay;
    }];
}

pandoc.nix

{haskellNix}:

haskell-nix.cabalProject {
  cabalProjectFreeze = null;
  cabalProject = null;
  cabalProjectLocal = null;

  compiler-nix-name = "ghc8104";
  name = "pandocProject";

  src = prev.fetchFromGitHub {
     name = "pandoc";
     owner = "jgm";
     repo = "pandoc";
     rev = "d05460d00d7c9af3b4913f1760ea385a7d855d84";
     sha256 = "1a3kwag6j13b42zhzxiwlzabsc6c9jppiwv9j8gbnf2k1yb84kdm";
  };
  pkg-def-extras = with final.haskell.lib; [];
}

Return



➜ nix build .#wikiPandoc
warning: Git tree '/home/reyman/TRASH/pandoc2nix' is dirty
error: flake output attribute 'legacyPackages.x86_64-linux.wikiPandoc' is not a derivation
(base)

I’m not sure that HaskellNix.overlay in pandoc.nix is well defined …

Any help is welcome, is there something i don’t do well @jonringer ?

My packages.x86_64-linux is empty {}

I also try this, without succes, i have an infinite recursion on (import ./nix/pandoc-overlay.nix)

pandoc-overlay.nix

{nixpkgs,haskellNix}:
final: prev: {
    pandocApp = final.haskell-nix.cabalProject {
      cabalProjectFreeze = null;
      cabalProject = null;
      cabalProjectLocal = null;

      compiler-nix-name = "ghc8104";
      name = "pandocProject";

      src = prev.fetchFromGitHub {
         name = "pandoc";
         owner = "jgm";
         repo = "pandoc";
         rev = "d05460d00d7c9af3b4913f1760ea385a7d855d84";
         sha256 = "1a3kwag6j13b42zhzxiwlzabsc6c9jppiwv9j8gbnf2k1yb84kdm";
      };
      pkg-def-extras = with final.haskell.lib; [];
    };
}

Flake.nix

{
  description = "PandocProject";
  inputs.haskellNix.url = "github:input-output-hk/haskell.nix";
  inputs.nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.wiki-python.url = "git+https://gitlab.huma-num.fr/gt-notebook/wiki.git?ref=main";

  outputs = { self, nixpkgs, flake-utils, haskellNix, wiki-python }:
      let
      my-overlay = nixpkgs.lib.composeManyExtensions [
        (import ./nix/pandoc-overlay.nix) # cause infinite recursion ...
        wiki-python.overlay
      ];

      pkgs = system: import nixpkgs {
          inherit system;
          overlays = [my-overlay];
      };

      in (flake-utils.lib.eachSystem ["x86_64-linux"] (system: rec {

      legacyPackages = pkgs system;

      packages = flake-utils.lib.flattenTree {
            inherit (legacyPackages overlay);
        };

        apps.myWiki = flake-utils.lib.mkApp { drv = legacyPackages.wikiApp; };
        apps.myPandoc = flake-utils.lib.mkApp { drv = packages.pandocApp; };

    }));
    }

Last but not last, i think i’m near the end but i don’t understand how to finalize correctly the haskell.nix part.

flake.nix :


{
  description = "PandocProject";
  inputs.haskell-nix.url = "github:input-output-hk/haskell.nix";
  inputs.nixpkgs.follows = "haskell-nix/nixpkgs-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.wiki-python.url = "git+https://gitlab.huma-num.fr/gt-notebook/wiki.git?ref=main";

  outputs = { self, nixpkgs, flake-utils, haskell-nix, wiki-python }:
      let
      my-overlay = nixpkgs.lib.composeManyExtensions [
      haskell-nix.overlay
      wiki-python.overlay  #(import ./nix/pandoc-overlay.nix)
          (final: prev: {
          pandocApp = final.haskell-nix.cabalProject {
          cabalProjectFreeze = null;
          cabalProject = null;
          cabalProjectLocal = null;

          compiler-nix-name = "ghc8104";
          name = "pandocProject";

          src = prev.fetchFromGitHub {
             name = "pandoc";
             owner = "jgm";
             repo = "pandoc";
             rev = "d05460d00d7c9af3b4913f1760ea385a7d855d84";
             sha256 = "1a3kwag6j13b42zhzxiwlzabsc6c9jppiwv9j8gbnf2k1yb84kdm";
          };
          pkg-def-extras = with final.haskell.lib; [];
          };}
          )

      ];

      pkgsForSystem = system:
        (import nixpkgs) {
          inherit system;
          overlays = [
            my-overlay
          ];
      };

      in (flake-utils.lib.eachSystem ["x86_64-linux"] (system: rec {

      pkgs = pkgsForSystem system;
      legacyPackages = pkgs;

      packages = flake-utils.lib.flattenTree {
            inherit (legacyPackages overlay);
        };


        #defaultPackage = packages.wp;
        apps.myWiki = flake-utils.lib.mkApp { drv = pkgs.wikiApp; };
        apps.myPandoc = flake-utils.lib.mkApp { drv = pkgs.pandocApp; };
    }));
    }

The result in nix repl :

nix-repl> apps.x86_64-linux
{ myPandoc = { ... }; myWiki = { ... }; }
nix-repl> apps.x86_64-linux.myPandoc 
trace: WARNING: 8.10.4 is out of date, consider using 8.10.7.
trace: No index state specified for pandocProject, using the latest index state that we know about (2021-11-22T00:00:00Z)!
trace: WARNING: 8.10.4 is out of date, consider using 8.10.7.
error: cannot coerce a set to a string

       at /nix/store/xfqi0z497g4qamqjlziaxfvzpc9f06kr-source/default.nix:136:18:

          135|       type = "app";
          136|       program = "${drv}${exePath}";
             |                  ^
          137|     };
nix-repl> apps.x86_64-linux.myWiki   
{ program = "/nix/store/4nmziia3qjdrgbrs4fvf7y89cgcxa7in-python3.9-notebook-wiki-0.0.1/bin/notebook-wiki"; type = "app"; }

This probably due to something in haskellNix overlay i don’t understand, in many example, like this https://github.com/hhefesto/respirare/blob/c44722f9cc6ee660b5b9a0baf4b4b521f12e056a/flake.nix, there is this part :

 [...]
      pkgs = import nixpkgs { inherit system overlays; };     
 flake = pkgs.respirare.flake {}; 
in flake // {
      # Built by `nix build .`
      defaultPackage = flake.packages."respirare:exe:respirare-exe";
[...]

Don’t know how to adapt this to my current flakes …

@reyman did you get any further with this ? RN I’m trying to get simplex-chat into my pkgs somehow (using flakes, simplex-chat relies on haskell.nix as well) and don’t know if worth a start ?