Announcing stacklock2nix: easily build a Haskell project that contains a `stack.yaml.lock` file

cdepillabout · November 29, 2022, 10:57am

I’ve just released a new Nix library: stacklock2nix

stacklock2nix converts a stack.yaml.lock file from a Haskell project into a Nixpkgs-compatible Haskell package set. This makes it (relatively) easy to build a Stack-based Haskell project with Nix.

stacklock2nix is an alternative to haskell.nix for Haskell projects that have a stack.yaml.lock file. stacklock2nix is a good candidate for introducing Nix into a project or team that is not already using Nix.

I’m using stacklock2nix to build a sizeable Haskell project at work, and it seems to be working relatively smoothly.

Here’s an example flake.nix (and a full example project) that you can use to get started playing around with stacklock2nix:

github.com

cdepillabout/stacklock2nix/blob/8b32b2aaf0c9d9a7c342be818b4e941b09067bbf/my-example-haskell-lib-easy/flake.nix#L39-L105

    
      
          # This is a top-level attribute that contains the result from calling
          # stacklock2nix.
          my-example-haskell-stacklock = final.stacklock2nix {
            stackYaml = ./stack.yaml;
          
          
  # The Haskell package set to use as a base.  You should change this
            # based on the compiler version from the resolver in your stack.yaml.
            baseHaskellPkgSet = final.haskell.packages.ghc924;
          
          
  # Any additional Haskell package overrides you may want to add.
            additionalHaskellPkgSetOverrides = hfinal: hprev: {
              # The servant-cassava.cabal file is malformed on GitHub:
              # https://github.com/haskell-servant/servant-cassava/pull/29
              servant-cassava =
                final.haskell.lib.compose.overrideCabal
                  { editedCabalFile = null; revision = null; }
                  hprev.servant-cassava;
            };
          
          
  # Additional packages that should be available for development.

This file has been truncated. show original

This snippet of Nix will build the Haskell project defined by this stack.yaml:

github.com

cdepillabout/stacklock2nix/blob/8b32b2aaf0c9d9a7c342be818b4e941b09067bbf/my-example-haskell-lib-easy/stack.yaml

# This file was automatically generated by 'stack init'
#
# Some commonly used options have been documented as comments in this file.
# For advanced use and comprehensive documentation of the format, please see:
# https://docs.haskellstack.org/en/stable/yaml_configuration/

# Resolver to choose a 'specific' stackage snapshot or a compiler version.
# A snapshot resolver dictates the compiler version and the set of packages
# to be used for project dependencies. For example:
#
# resolver: lts-3.5
# resolver: nightly-2015-09-21
# resolver: ghc-7.10.2
#
# The location of a snapshot can be provided as a file or url. Stack assumes
# a snapshot provided as a file might change, whereas a url resource does not.
#
# resolver: ./custom-snapshot.yaml
# resolver: https://example.com/snapshots/2018-01-01.yaml
resolver: nightly-2022-10-18

This file has been truncated. show original

If you’re new to Nix, you’ll also want to read the Quickstart section of the stacklock2nix README to learn what commands to run to build the project.

NobbZ · November 29, 2022, 11:25am

As my biggest pain point with Haskell in nix is the quasi requirement of IFD, does your project work without IFD or can it at least be used (easily) to pregenerate a couple of expressions that then can be used without IFD?

cdepillabout · November 30, 2022, 12:02am

stacklock2nix does rely on IFD. It uses IFD in a few places:

Reading .yaml files into Nix. stacklock2nix reads stack.yaml and stack.yaml.lock in as Nix values. To do this, it needs to convert them to .json files and call builtins.readFile and builtins.fromJSON on them. (This wouldn’t be necessary if Nix had a builtins.fromYAML function, or stacklock2nix used something like fromYaml/fromYaml.nix at 3d4282fd9bbc0628e05d7da18b49d3ee0fa7e8fd · DavHau/fromYaml · GitHub)
Recursively reading Stackage resolvers. In most projects, the stack.yaml.lock points to a Stackage resolver, which is also a .yaml file hosted on GitHub. This has to be downloaded and then read in as Nix, using IFD.
stacklock2nix uses functions like callHackage and callCabal2nix under the hood. These do IFD.

It can’t. I’m guessing you’d like a project like stack2nix, which can be used to generate a bunch of .nix files that contain a Haskell package set overlay. Unfortunately, stack2nix is no longer maintained.

haskell.nix might also have functionality for generating a bunch of .nix files that can be used without IFD, but I’ve never personally used this functionality.

What makes this a pain point for you?

In most of the project I’ve worked in, people generally want a single source of truth with regards to build inputs. In the case of Haskell projects, this is either:

a stack.yaml and stack.yaml.lock file

OR

some combination of a cabal.project file, cabal.freeze file, Cabal solver version, and Hackage generate date

along with .cabal files for each local package. People generally don’t want to commit .nix files into the repo that duplicate this type of information. Especially when you have to remember to update these files by hand (or setup some sort of automation for this).

When talking to people about this, I’ve never personally heard anyone say that IFD is a pain-point when dealing with Haskell in Nix. Although I have frequently heard that not having a single source of truth is quite annoying. What makes IFD a pain point for you?

573 · November 30, 2022, 10:04am

Thanks for sharing another awesome project with us.

I’m into learning haskell for quite a while now, with a special interest in building static binaries and can’t wait to get my hands on stacklock2nix.

What @NobbZ refers to are IMHO mainly two things:

Lockfile-to-builder packages - #7 by NobbZ - longer evaluation times and
Deduplicating a package definition between nixpkgs and a Flake - #4 by NobbZ - fail fast on errors

NobbZ · November 30, 2022, 10:38am

It is not “fail fast on errors”, it is that IFD doesn’t suite flake tooling well.

As soon as a flakes output uses IFD and produces any output which’s target system isn’t the one that evaluates the expression it breaks.

This is somethings that can be fixed in nix, as shown in the PR linked from the second link.

I know that I am relatively alone with my strict approach to IFD, though lets be honest…

My setting of strictly forbidding IFD, does not impact me in daily use, unless I touch haskell.

In other situations it even uncovered bugs that I would probably not have seen otherwise. Most recent one was when I inheritted something that I shouldn’t inherit because of a typo.

I understand how people do not want to have nix files “poluting” their clean projects. I understand the demand for a single source of truth.

So: convert the lockfiles into JSON in repo. No need for IFD anymore. Use projects like the cabal hashes in JSON format in the nix-community org. Or even fix the haskell tooling to use a lockfile and hashing format we can actually use directly.

What annoys me most:

There is no maintained alternative, that would allow to not use IFD easily. Someone suggested to use haskell.nix already, though that again seems to include an hour long ceremony to regenerate the haskell lock files on each change in the cabal file.

Damn, other ecosystems have a oneshot command one can use to pregenerate the files just in the repo, making it easy to use them in the “pregenerate” or “IFD” modes.

Just think abobut it… How easy could it be, if there was a oneshot command that generates a nix compatible lock file that just can be consumed from within nixpkgs, just passing it to a builder along the source.

I really hope that dream2nix will be there soon, and I wish, there would be more contributions to it rather than fractioning a single languages ecosystem even more.

I’m really thankful for contributions to any of the ecosystems, though I fail to see the value in creating yet another project, that just works the same than the previous ones.

Don’t get me wrong, I’m pretty sure there was a reason why @cdepillabout felt the need to implement yet another build tool for Haskell, and its probably valuable for that use case, but what makes that usecase not a good fit for the already existing tooling? Why hasn’t the existing tooling got adjusted to match that use case?

TL’DR: My biggest problem with IFD, it is just assumed in the haskell ecosystem. There is no easy way to opt out for those who do not want to use it. The reasons for their desire are not relevant. Perhaps its just because they want to use a hydra instance for CI…

PS: I am totally fine to avoid haskell until there are alternatives, still I will continue to ask the same question everytime a now tool pops up.

domenkozar · November 30, 2022, 11:24am

Nice! What would be amazing if this project is ported to dream2nix as that allows us to unify language support.

cdepillabout · November 30, 2022, 11:25am

Let me try to respond to some of your points.

If your language / build system has lockfiles in some format other than JSON, then converting them into JSON would either require IFD (similar to what stacklock2nix is doing), or you’d no longer have a single source of truth.

Although you do have a good point. If your lockfile is in JSON or TOML, it is easy to interpret with builtins.fromJSON or builtins.fromTOML.

I guess you’re talking about: GitHub - nix-community/all-cabal-json at hackage

The problem is that if you want the solution to work with an arbitrary Haskell package, you need a way to take a .cabal file as input, and parse out the dependency information you need. There are various approaches to this, like cabal2nix, haskell.nix, cabal2json, and cabal2nixWithoutIFD. Its not a simple problem.

fix the haskell tooling to use a lockfile and hashing format we can actually use directly.

I can only imagine how much work it would be to try to convince the entire Haskell community that JSON or TOML should solely be used as input to builds… (although there is actually a cabal issue about this: Migrate from the .cabal format to a widely supported format · Issue #7548 · haskell/cabal · GitHub)

There is no existing tooling for taking a Haskell project with a stack.yaml.lock file and turning it into a Nixpkgs-compatible Haskell overlay. If there had been, then I wouldn’t have needed to write stacklock2nix!

(There may be something available in dream2nix, but then I’m stuck with using dream2nix. Not that that may be problem in the long run if dream2nix gets really popular, but for now I’d rather have a lower-level tool like stacklock2nix that dream2nix could use.)

Why hasn’t the existing tooling got adjusted to match that use case?

That’s a good question. My guess is that both Nix and Haskell are relatively niche, so the total number of people working on tooling is just not that large. A lot of people are happy using haskell.nix, even though as you say, it can be somewhat slow.

You may very well be aware of this, but I think Hydra does allow IFD (at least that’s how it was explained to me). Like, if you run your own Hydra, you can set it up to allow IFD. Its just the NixOS Hydra that doensn’t allow IFD.

Your point still stands though. Someone might want to use a non-IFD approach to build a Haskell project in Nixpkgs.

Currently all we have is cabal2nix. It is possible to use cabal2nix to build a Haskell package without IFD (and even well-documented in GitHub - Gabriella439/haskell-nix: Nix and Haskell in production), but it isn’t particularly suited for use with a whole Haskell project (based on the cabal.project + the Cabal solver, or Stackage resolvers).

It does seem like it would be an interesting feature to add to stacklock2nix to try to replicate what the old stack2nix project did (with creating a full .nix file that could be used without IFD). If you wanted to open an issue for this, we could discuss it more.

cdepillabout · November 30, 2022, 11:30am

I thought it would be nice to setup stacklock2nix so that it could be used as a lower-level library that dream2nix could depend on and call into.

I’m not sure exactly what sort of API stacklock2nix would need to expose for this, but if someone wanted to create an issue for this, I’m happy to discuss it further.

NobbZ · November 30, 2022, 12:21pm

In case of haskell it wouldn’t be a problem. JSON is a subset of YAML, and any YAML parser should accept the JSON.

I see its an inconvinience to have to remember constantly “downgrading” the generated YAML.

cdepillabout · November 30, 2022, 12:31pm

In the case of Haskell, even if you converted the stack.yaml and stack.yaml.lock files to JSON, the stack.yaml.lock file would still reference an upstream resolver file that is in YAML.

You could do something like in-lining the resolver directly into your stack.yaml.lock file, but at that point you’re sort of stepping outside the easy path that stack has laid out for you.

And then you still have the problem of converting your local .cabal files into JSON.

DavHau · December 3, 2022, 8:40am

If there is a limited number of stack resolvers we could pre-convert them all to JSON and re-host on github.
… or we just work together moving forward the builtins.fromYAML PR. It could use some reviews.

Concerning integration with dream2nix, I might open a github issue to not pollute this discussion here too much.

Dream2nix is currently designed to guarantee a certain feature set for all supported ecosystems. One example being the option to render your inputs to a JSON compatible lock, so that no IFD will be required after that.

This will basically require you to split your lang2nix tool into two parts. One that produces the lock and another one that consumes the lock. Doing that can require non-trivial refactoring of your tool.

It has been suggested already by multiple people to open up dream2nix a bit more and allow it to just act as frontend for various existing tools, without requiring them to change their architecture.
I am seriously considering this, but the downside of this would then be that different subsystems in dream2nix would support a different set of features, potentially downgrading the overall UX. Not an easy decision to make.

cdepillabout · December 3, 2022, 9:29am

In theory, someone could make up their own package set into their own Stackage resolver. But I’ve never actually seen someone do this in practice.

So we could convert all Stackage resolvers to JSON and rehost on GitHub. I thought that haskell.nix actually did this, but I couldn’t find the repo after a little searching. Maybe someone else knows the repo I’m thinking of and would be able to link it. Or maybe I’m just misremembering.

That’s pretty interesting. Hopefully that gets merged in.

I think this is pretty smart, and I’d be happy to make any changes to stacklock2nix to support generating a JSON file like this. If you make an issue, please feel free to ping me. Or alternatively open an issue on the stacklock2nix repo as well: Issues · cdepillabout/stacklock2nix · GitHub

I imagine some changes would also need to be made to the underlying Haskell tools (like cabal2nix), but maybe whatever is required has already been implemented in dream2nix.

pimiddy · December 4, 2022, 9:11am

Sorry for the stupid question, but what’s IFD?

NobbZ · December 4, 2022, 9:37am

https://nixos.wiki/wiki/Import_From_Derivation

cdepillabout · December 15, 2022, 1:14pm

I started writing a blog post series about examples of building a few different Haskell projects with Nix:

https://functor.tokyo/blog/2022-12-15-stacklock2nix

The first post is about building the PureScript compiler with stacklock2nix:

https://functor.tokyo/blog/2022-12-16-building-purescript-with-stacklock2nix

DavHau · December 16, 2022, 7:14am

I opened a WIP PR that initializes teamplates + tests for a dream2nix integration.
I do not intend to complete this myself, but it can serve as a starting point and a place for discussions.

github.com/cdepillabout/stacklock2nix

init dream2nix integration templates + tests

cdepillabout:main ← DavHau:dream2nix

opened 07:12AM - 16 Dec 22 UTC

DavHau

+436 -2

This PR initializes templates for integration with dream2nix and adds a test tha…t can be executed via: `nix run .#dream2nix-integration-test`. I do not intend to complete this PR, but it can serve as a starting point for whoever wants to take on the work, and can be used for discussions. The templates for the translator and builder are largely just copied from `dream2nix/src/templates/...`, and contain only minor modifications. Usually we commit the modules for ecosystem support like these, directly to the dream2nix source itself, but as I understood you, @cdepillabout, the intention is to keep the stacklock2nix project separate. Therefore the PR adds the dream2nix modules to the stacklock2nix project itself and exposes them via the flake outputs. Feel free anytime to move this whole project inside dream2nix and maintain it from there, if you feel comfortable with that. Some comments in the templates contain references to other files, like examples or specifications. Those references are relative to the source of the upstream dream2nix project. In general, some more detailed guide on contributing ecosystem support to dream2nix can be found [here](https://nix-community.github.io/dream2nix/contributing/contributing.html) ## Short Summary The purpose of the translator is to extract data from any given project source (by parsing the stack.yaml.lock for example) and return the extracted data via the dream-lock format, which is JSON serializable. The specification of the dream-lock format can be found at [dream2nix/src/specifications](https://github.com/nix-community/dream2nix/tree/main/src/specifications) A successful termination of the .#dream2nix-integration-test app implies that the dream-lock format produced by the translator is correct. The purpose of the builder is to read the data represented by the dream-lock and generate nix `packages` and `devShells` from it. The builder accesses the dream-lock indirectly through getter functions like `getDependencies` etc. which are provided by dream2nix (see builder template below). Now there is a clean way and a dirty way of integrating stacklock2nix into dream2nix. The dream-lock format contains fields like `dependencies` and `sources`, for which the format is strict. If those fields are used, it will: - allow dream2nix to take care of fetching dependencies - allow dream2nix to provides interfaces like `inject` and `sourceOverrides` allowing the user to manipulate the the dependency tree and sources - ensure compatibility between different translators and builders for haskell in dream2nix Besides these strict fields, the dream-lock also offers the `_subsystem` field, which can be used as a free-form field. So the quick and dirty way would be to ignore some fields of the dream-lock and instead put data into `_subsystem`. That would work around some of the requirements of dream2nix but kill some features and general compatibility with other haskell modules. It would be great to do it the clean way, so we can re-use the stacklock2nix builder for example with other translators like `cabal-plan` etc. and we won't have to do the work twice. Feel free to comment if you have any more questions.

ParetoOptimalDev · December 16, 2022, 7:57pm

I think this problem will become larger as mindshare around me and in general seems to be moving away from stack to cabal.

I personally abandoned the single source of truth approach using haskell.nix for haskell4nix and manually keeping cabal.project and flake.nix in sync with about 50 overrides lol.

cdepillabout · December 19, 2022, 1:09pm

The second post in my blog series is up:

https://functor.tokyo/blog/2022-12-20-building-dhall-with-stacklock2nix

This is about building all the Dhall tools (written in Haskell) with stacklock2nix. The Dhall project is a little more complicated than PureScript, and shows off the full power off stacklock2nix.

cdepillabout · December 26, 2022, 11:54pm

The third and final post in my blog series is online:

https://functor.tokyo/blog/2022-12-26-building-pandoc-with-stacklock2nix

This post is about building Pandoc with stacklock2nix. This is similar to the first post about building the PureScript compiler, but this time I’ve setup Pandoc to be fully statically-linked.

This is nice if you want to build a Haskell binary with Nix, but use it on a system without Nix.

domenkozar · December 27, 2022, 10:20am

Is it possible not to use the base package set from nixpkgs and rather generate it from stackage?

I’m afraid that some override will be pulled in from nixpkgs that will differ between development (stack.yaml) and production (nix).