Announcing stacklock2nix: easily build a Haskell project that contains a `stack.yaml.lock` file

Announcements
1

I’ve just released a new Nix library: stacklock2nix

stacklock2nix converts a stack.yaml.lock file from a Haskell project into a Nixpkgs-compatible Haskell package set. This makes it (relatively) easy to build a Stack-based Haskell project with Nix.

stacklock2nix is an alternative to haskell.nix for Haskell projects that have a stack.yaml.lock file. stacklock2nix is a good candidate for introducing Nix into a project or team that is not already using Nix.

I’m using stacklock2nix to build a sizeable Haskell project at work, and it seems to be working relatively smoothly.

Here’s an example flake.nix (and a full example project) that you can use to get started playing around with stacklock2nix:

This snippet of Nix will build the Haskell project defined by this stack.yaml:

If you’re new to Nix, you’ll also want to read the Quickstart section of the stacklock2nix README to learn what commands to run to build the project.

1 Like
2

As my biggest pain point with Haskell in nix is the quasi requirement of IFD, does your project work without IFD or can it at least be used (easily) to pregenerate a couple of expressions that then can be used without IFD?

1 Like
3

stacklock2nix does rely on IFD. It uses IFD in a few places:

  1. Reading .yaml files into Nix. stacklock2nix reads stack.yaml and stack.yaml.lock in as Nix values. To do this, it needs to convert them to .json files and call builtins.readFile and builtins.fromJSON on them. (This wouldn’t be necessary if Nix had a builtins.fromYAML function, or stacklock2nix used something like https://github.com/DavHau/fromYaml/blob/3d4282fd9bbc0628e05d7da18b49d3ee0fa7e8fd/fromYaml.nix)

  2. Recursively reading Stackage resolvers. In most projects, the stack.yaml.lock points to a Stackage resolver, which is also a .yaml file hosted on GitHub. This has to be downloaded and then read in as Nix, using IFD.

  3. stacklock2nix uses functions like callHackage and callCabal2nix under the hood. These do IFD.

It can’t. I’m guessing you’d like a project like stack2nix, which can be used to generate a bunch of .nix files that contain a Haskell package set overlay. Unfortunately, stack2nix is no longer maintained.

haskell.nix might also have functionality for generating a bunch of .nix files that can be used without IFD, but I’ve never personally used this functionality.

What makes this a pain point for you?

In most of the project I’ve worked in, people generally want a single source of truth with regards to build inputs. In the case of Haskell projects, this is either:

  • a stack.yaml and stack.yaml.lock file

OR

  • some combination of a cabal.project file, cabal.freeze file, Cabal solver version, and Hackage generate date

along with .cabal files for each local package. People generally don’t want to commit .nix files into the repo that duplicate this type of information. Especially when you have to remember to update these files by hand (or setup some sort of automation for this).

When talking to people about this, I’ve never personally heard anyone say that IFD is a pain-point when dealing with Haskell in Nix. Although I have frequently heard that not having a single source of truth is quite annoying. What makes IFD a pain point for you?

4

Thanks for sharing another awesome project with us.

I’m into learning haskell for quite a while now, with a special interest in building static binaries and can’t wait to get my hands on stacklock2nix.

What @NobbZ refers to are IMHO mainly two things:

1 Like
5

It is not “fail fast on errors”, it is that IFD doesn’t suite flake tooling well.

As soon as a flakes output uses IFD and produces any output which’s target system isn’t the one that evaluates the expression it breaks.

This is somethings that can be fixed in nix, as shown in the PR linked from the second link.

I know that I am relatively alone with my strict approach to IFD, though lets be honest…

My setting of strictly forbidding IFD, does not impact me in daily use, unless I touch haskell.

In other situations it even uncovered bugs that I would probably not have seen otherwise. Most recent one was when I inheritted something that I shouldn’t inherit because of a typo.

I understand how people do not want to have nix files “poluting” their clean projects. I understand the demand for a single source of truth.

So: convert the lockfiles into JSON in repo. No need for IFD anymore. Use projects like the cabal hashes in JSON format in the nix-community org. Or even fix the haskell tooling to use a lockfile and hashing format we can actually use directly.

What annoys me most:

There is no maintained alternative, that would allow to not use IFD easily. Someone suggested to use haskell.nix already, though that again seems to include an hour long ceremony to regenerate the haskell lock files on each change in the cabal file.

Damn, other ecosystems have a oneshot command one can use to pregenerate the files just in the repo, making it easy to use them in the “pregenerate” or “IFD” modes.

Just think abobut it… How easy could it be, if there was a oneshot command that generates a nix compatible lock file that just can be consumed from within nixpkgs, just passing it to a builder along the source.

I really hope that dream2nix will be there soon, and I wish, there would be more contributions to it rather than fractioning a single languages ecosystem even more.

I’m really thankful for contributions to any of the ecosystems, though I fail to see the value in creating yet another project, that just works the same than the previous ones.

Don’t get me wrong, I’m pretty sure there was a reason why @cdepillabout felt the need to implement yet another build tool for Haskell, and its probably valuable for that use case, but what makes that usecase not a good fit for the already existing tooling? Why hasn’t the existing tooling got adjusted to match that use case?

TL’DR: My biggest problem with IFD, it is just assumed in the haskell ecosystem. There is no easy way to opt out for those who do not want to use it. The reasons for their desire are not relevant. Perhaps its just because they want to use a hydra instance for CI…

PS: I am totally fine to avoid haskell until there are alternatives, still I will continue to ask the same question everytime a now tool pops up.

3 Likes
6

Nice! What would be amazing if this project is ported to dream2nix as that allows us to unify language support.

4 Likes
7

Let me try to respond to some of your points.

If your language / build system has lockfiles in some format other than JSON, then converting them into JSON would either require IFD (similar to what stacklock2nix is doing), or you’d no longer have a single source of truth.

Although you do have a good point. If your lockfile is in JSON or TOML, it is easy to interpret with builtins.fromJSON or builtins.fromTOML.

I guess you’re talking about: GitHub - nix-community/all-cabal-json at hackage

The problem is that if you want the solution to work with an arbitrary Haskell package, you need a way to take a .cabal file as input, and parse out the dependency information you need. There are various approaches to this, like cabal2nix, haskell.nix, cabal2json, and cabal2nixWithoutIFD. Its not a simple problem.

fix the haskell tooling to use a lockfile and hashing format we can actually use directly.

I can only imagine how much work it would be to try to convince the entire Haskell community that JSON or TOML should solely be used as input to builds… (although there is actually a cabal issue about this: Migrate from the .cabal format to a widely supported format · Issue #7548 · haskell/cabal · GitHub)

There is no existing tooling for taking a Haskell project with a stack.yaml.lock file and turning it into a Nixpkgs-compatible Haskell overlay. If there had been, then I wouldn’t have needed to write stacklock2nix!

(There may be something available in dream2nix, but then I’m stuck with using dream2nix. Not that that may be problem in the long run if dream2nix gets really popular, but for now I’d rather have a lower-level tool like stacklock2nix that dream2nix could use.)

Why hasn’t the existing tooling got adjusted to match that use case?

That’s a good question. My guess is that both Nix and Haskell are relatively niche, so the total number of people working on tooling is just not that large. A lot of people are happy using haskell.nix, even though as you say, it can be somewhat slow.

You may very well be aware of this, but I think Hydra does allow IFD (at least that’s how it was explained to me). Like, if you run your own Hydra, you can set it up to allow IFD. Its just the NixOS Hydra that doensn’t allow IFD.

Your point still stands though. Someone might want to use a non-IFD approach to build a Haskell project in Nixpkgs.

Currently all we have is cabal2nix. It is possible to use cabal2nix to build a Haskell package without IFD (and even well-documented in GitHub - Gabriella439/haskell-nix: Nix and Haskell in production), but it isn’t particularly suited for use with a whole Haskell project (based on the cabal.project + the Cabal solver, or Stackage resolvers).

It does seem like it would be an interesting feature to add to stacklock2nix to try to replicate what the old stack2nix project did (with creating a full .nix file that could be used without IFD). If you wanted to open an issue for this, we could discuss it more.

1 Like
8

I thought it would be nice to setup stacklock2nix so that it could be used as a lower-level library that dream2nix could depend on and call into.

I’m not sure exactly what sort of API stacklock2nix would need to expose for this, but if someone wanted to create an issue for this, I’m happy to discuss it further.

2 Likes
9

In case of haskell it wouldn’t be a problem. JSON is a subset of YAML, and any YAML parser should accept the JSON.

I see its an inconvinience to have to remember constantly “downgrading” the generated YAML.

3 Likes
10

In the case of Haskell, even if you converted the stack.yaml and stack.yaml.lock files to JSON, the stack.yaml.lock file would still reference an upstream resolver file that is in YAML.

You could do something like in-lining the resolver directly into your stack.yaml.lock file, but at that point you’re sort of stepping outside the easy path that stack has laid out for you.

And then you still have the problem of converting your local .cabal files into JSON.

2 Likes
11

If there is a limited number of stack resolvers we could pre-convert them all to JSON and re-host on github.
… or we just work together moving forward the builtins.fromYAML PR. It could use some reviews.

Concerning integration with dream2nix, I might open a github issue to not pollute this discussion here too much.

Dream2nix is currently designed to guarantee a certain feature set for all supported ecosystems. One example being the option to render your inputs to a JSON compatible lock, so that no IFD will be required after that.

This will basically require you to split your lang2nix tool into two parts. One that produces the lock and another one that consumes the lock. Doing that can require non-trivial refactoring of your tool.

It has been suggested already by multiple people to open up dream2nix a bit more and allow it to just act as frontend for various existing tools, without requiring them to change their architecture.
I am seriously considering this, but the downside of this would then be that different subsystems in dream2nix would support a different set of features, potentially downgrading the overall UX. Not an easy decision to make.

5 Likes
12

In theory, someone could make up their own package set into their own Stackage resolver. But I’ve never actually seen someone do this in practice.

So we could convert all Stackage resolvers to JSON and rehost on GitHub. I thought that haskell.nix actually did this, but I couldn’t find the repo after a little searching. Maybe someone else knows the repo I’m thinking of and would be able to link it. Or maybe I’m just misremembering.

That’s pretty interesting. Hopefully that gets merged in.

I think this is pretty smart, and I’d be happy to make any changes to stacklock2nix to support generating a JSON file like this. If you make an issue, please feel free to ping me. Or alternatively open an issue on the stacklock2nix repo as well: Issues · cdepillabout/stacklock2nix · GitHub

I imagine some changes would also need to be made to the underlying Haskell tools (like cabal2nix), but maybe whatever is required has already been implemented in dream2nix.

4 Likes
13

Sorry for the stupid question, but what’s IFD?

3 Likes
14

https://nixos.wiki/wiki/Import_From_Derivation

4 Likes
15

I started writing a blog post series about examples of building a few different Haskell projects with Nix:

The first post is about building the PureScript compiler with stacklock2nix:

3 Likes
16

I opened a WIP PR that initializes teamplates + tests for a dream2nix integration.
I do not intend to complete this myself, but it can serve as a starting point and a place for discussions.
https://github.com/cdepillabout/stacklock2nix/pull/8

2 Likes
17

I think this problem will become larger as mindshare around me and in general seems to be moving away from stack to cabal.

I personally abandoned the single source of truth approach using haskell.nix for haskell4nix and manually keeping cabal.project and flake.nix in sync with about 50 overrides lol.

1 Like
18

The second post in my blog series is up:

This is about building all the Dhall tools (written in Haskell) with stacklock2nix. The Dhall project is a little more complicated than PureScript, and shows off the full power off stacklock2nix.

1 Like
19

The third and final post in my blog series is online:

This post is about building Pandoc with stacklock2nix. This is similar to the first post about building the PureScript compiler, but this time I’ve setup Pandoc to be fully statically-linked.

This is nice if you want to build a Haskell binary with Nix, but use it on a system without Nix.

20

Is it possible not to use the base package set from nixpkgs and rather generate it from stackage?

I’m afraid that some override will be pulled in from nixpkgs that will differ between development (stack.yaml) and production (nix).

1 Like