Strong opinion: Library packaging

Development
4

I think we’re thinking along similar lines. The idea of a lingua franca for application dependencies is a really excellent one. @samueldr’s point about Python hacking (nix-shell -p ...) is cogent as well, I think.

So, here’s what I’d like to propose:

Two general types of expression. (I don’t know if they need to be formal types or just “the set accepted by this function”). First, something like haskellPackages or python.modules - a set of sets, a la

{ 
  libraryname = {
    "13.17.1": {
      source = "github",
      sha256 = "...",
      #...,
      buildInputs = [libzip, sqlite, libxml2];
    };
    #...
  };
  otherLib = {
    "20180530" = {
      source = "langrepo",
      sha256 = "...",
    }
  #...
};

Notably, languagePackages shouldn’t care about the other-library dependencies of libraries.

Based on the experiences of Python packagers, each set of library modules should probably be a sub-tree of files, each imported into the top level packages set, as a hedge against merge collisions.

Second, a generalization of defaultGemConfig, to allow overrides of the derivations produced by the languagePackages. The nokogiri attribute from @manveru’s example is perfect.

From there, applications assemble their “lockfile” by doing something like:

stdenv.mkDerivation {
  #...
  buildInputs = [ postgres ] + 
    (langInputs language.modules [ "libraryname-13.17.1" ]);
};

The nice thing about all this is that I think it can all be generated automatically. We leave it to per-language tools to satisfy dependency files and produce lockfiles. The tools already produced and in use by the language communities should be used, and any gaps filled on our side.

In this scheme, a tool like bundix would examine Gemfile.lock, conceivably running bundle lock under the covers to generate it if needed (as it does when run as bundix -m). The result would be a list of strings (names of gems with versions, in this case) in a lockfile.nix suitable for import. In the same operation, bundix would consult the ruby.modules set and see which members of the lockfile are already provided, and produce a separate gemset.nix set of the missing items.

The idealistic result, then, would be that an in-development expression for a Ruby app would look like:

stdenv.mkDerivation {
  #...
  buildInputs = rubyInputs (ruby.modules // import ./extraset.nix) (import ./lockfile.nix);
}

and that the expression for a Haskell app would look like:

stdenv.mkDerivation {
  #...
  buildInputs = haskellInputs (haskell.modules // import ./extraset.nix) (import ./lockfile.nix);
}

I think there’s a lot of potential to something like this, including standardizing tooling and naming around it, reduced complexity for onboarding and transferabillity of skill across languages within Nix.

1 Like