# State of the BEAM Ecosystem in Nix
**URL:** https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202
**Category:** Development
**Created:** [September 29, 2019, 3:15pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202 "2019-09-29T15:15:25Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [September 29, 2019, 3:15pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/1 "2019-09-29T15:15:25Z")
Hello,
My team recently begun using Nix/NixOS as our main packager, OS and mechanism for deployment for our Elixir projects. We have a growing code base in BEAM languages, and we would like to use Nix more.
Making it work in Nix has been a journey, and now that we got there, we would like to help streamline that a bit upstream. We are ready to spent time and energy doing so. We have some ideas, but we would like that to be a community effort. There have been interest in Nix in the BEAM ecosystem, but the current state of the BEAM tooling in nix is not particularly helping convinced people through.
I am going to make this a 2 parts thing. First around the state of the ecosystem, then around the ideas we have. This is a WIP after some discussion with people that were active in the BEAM part of nixpkgs. I fully expect this to be a limited and partial view of the state of the ecosystem and to learn a lot from the community. It is meant to be a place to coordinate and organize before working on significant change.
If you see anything you disagree with or that seems factually incorrect… That is why i make this post ! Please tell us about it 🙂
## State of the BEAM Ecosystem
- `pkgs/top-level/beam-packages.nix` provide some named version of erlang under `beam.interpreters`. Among others, we find version with odbc, with the javac flags and some `_nox` version that are headless version without the graphic environement brought through the `:observer` tool. There is also the (probably now unmaintained) Basho fork of R16 for Riak. Some of these versions are also combined to offer elixir and LFE interpreters.
- `pkgs/top-level/beam-packages.nix` also provides a `packagesWith ` helper to generate a package set with a specific version of erlang, see below for the meaning of the package set.
- `pkgs/development/beam-modules` provides a ton of things. Deep down, it is meant to act as an interface with the BEAM community tooling, but i do not think it is well maintained or used by the community at this time.
- Mix and Erlang.mk “works as expected”. This means that when used as expected by the BEAM community, they will not work in sandboxed mode. They also have a “bootstrap” step that need to be run before using them that is not well documented in what it does or why. (which is related to `buildMix` and `buildErlangMk`
- BEAM packages mostly come from Hex. This is the package registry the BEAM community has converged on. There is a semi automatic tool to run them and copy builders for them into the nixpkgs repository in a snapshot. This seems to not be maintained, nor heavily used.
- Rebar3 is offered in two version. One of them is hermetic `rebar3` and the other is the normal Rebar3 called `rebar3-open`. This seems anterior to the widespread use of sandboxing.
- There is a `buildHex` and a `fetchHex` function that allows for direct use of Hex package name and SHA256. That should work with sandboxing.
- `fetch-rebar-deps` does not work outside of sandboxing.
- Deep down, the way to use Nix tooling seems to be that we need to handwrite all the dependencies in the nix derivation. This is kinda opposite to the current situation of both rebar3 and Mix, which use lockfile of their own, with proper hash, to define the dependencies.
- @Ankhers has been working on a tool that can parse a rebar lockfile and translate it into a nix file that could be imported.
## Way forward
Here are some propositions i think may make sense. The first two i think are better done sooner than later. They have high maintenance burden that are not done right now for very few benefit and they lock us for exploring better solutions.
- Get rid of the hermetic rebar3. Rename `rebar3-open` to `rebar3`". We have sandboxing by default now anyway.
- Get rid of the whole snapshot package set. This is not well maintained anyway. And `fetchHex` can use Hex directly.
- Explore simplifying the way we build elixir and LFE now that we do not have to maintain the package set
- Explore cleaning up the bootstrapping part for mix. If necessary work with the mix team to simplify it.
- Update the version of Hex in nixpkgs.
- Bring @Ankhers tool (`rebar32nix`) in and provide an equivalent `mix2nix`. Explore ways to build an intermediate derivation with it, or just import a `dep.nix` generated with it.
- Update the documentation accordingly
- Explore the possibility to support the `scope` feature of nixpkgs.
- [New] Fix the doubling of `erl/libs` in the path, producing a lot of double load warning in Elixir.
## Previous Work:
State of the BEAM ecosystem : [[Suggestion] State of the Erlang ecosystem in NixOS · Issue #53834 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/issues/53834)
@Ankhers’s `rebar32nix` : [GitHub - ankhers/rebar32nix · GitHub](https://github.com/ankhers/rebar32nix)
Possibly interested people:
@Ankhers, @hauleth, @DianaOlympos
---
### Author:  [@anon31783435](https://discourse.nixos.org/u/anon31783435)
#### Post date: [September 30, 2019, 11:09am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/2 "2019-09-30T11:09:33Z")
I think a lot of this is related to the following discussion: [Github Issue](https://github.com/NixOS/nixpkgs/issues/53834)
Get rid of the hermetic rebar3: Should be already done? [Source](https://github.com/NixOS/nixpkgs/pull/54115/files)
I appreciate this discussion very much, but I can only talk about the Elixir related things.
- The snapshot package set is in my opinion really not the way to go.
- Doubling of `erl/libs` in the path is really annoying.
On my side as I mainly use Nix to set-up my Elixir development environment, I had a lot of thoughts if I should convert my mix deps to a nix deps declaration or if I just want to get my deps via `mix` as they are already determined do a fixed version by my `mix.lock`
I came to the conclusion I have no desire to convert them to `nix`. I just want to simply do a `mix deps.get` to setup my development environment.
That this needs Nix to run with `sandbox = false` is sadly far away from obvious.
Everyone who wants to get a picture of how I’m using Nix with Elixir: [Github](https://github.com/cw789/elixir_nix_seed)
PS: Reasons why I use Nix with Elixir:
- Ability to pin Elixir (Erlang) version for every developer of my repository
- Caching of the build layers within CI (not as efficient possible with any other technology)
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [September 30, 2019, 12:08pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/3 "2019-09-30T12:08:37Z")
For rebar3: it seems we need to update the docs then. Will look at this tonight
For the mix.lock i agree. The sandboxing false could be documented, and we could probably document better the whole BEAM part, with an “operator cookbook” mindset.
If we had a mix2nix, we could allow people to use nix normally in dev and build with nix in prod.
Thanks for the github link, i will have a look
---
### Author:  [@hkochev](https://discourse.nixos.org/u/hkochev)
#### Post date: [October 5, 2019, 2:55pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/4 "2019-10-05T14:55:02Z")
actually there 2 mix2nix packages:
> **[GitHub - georgewhewell/mix2nix](https://github.com/georgewhewell/mix2nix)**
>
> Contribute to georgewhewell/mix2nix development by creating an account on GitHub.
> **[Transumption / Mix to Nix · GitLab](https://gitlab.com/transumption/mix-to-nix)**
>
> Sandboxable mix.lock and rebar.lock Nix generator
---
### Author:  [@manveru](https://discourse.nixos.org/u/manveru)
#### Post date: [October 5, 2019, 4:44pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/5 "2019-10-05T16:44:34Z")
I also wrote [Michael Fellinger / mixnix · GitLab](https://gitlab.com/manveru/mixnix) some time ago, not using Elixir anymore though so it’s been kind of stale.
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [October 6, 2019, 11:16am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/6 "2019-10-06T11:16:42Z")
So it seems noone is really trying to offer a defense of the current BEAM packages set. I will open an issue on nixos to get rid of it and update documentation.
@hkochev and @manveru thank you so much, lot of interesting stuff in these tools. It seems to me that the best direction for the “2nix” tools needs more debate and work. I will ping @Ankhers again for his work and i would encourage everyone to offer options.
The options I see so far are
1. Use an escript that take the `.lock` and output a `deps.nix` to import by the user
2. Generate a temporary JSON from mix.lock through a small escript and then use it to generate fetchURL and fetchGit calls on the fly.
1 may be the easier to produce right now, as it can be build easily, but it asks the user to rerun the script everytime they change the mix.lock.
2 integrate really nicely into the normal nix packaging, but it needs bootstrapping and may be a bit more opaque.
In the meantime, we could offer something like Go, and use the non sandbox classic “getting deps” of the tooling in the `build` tools. I am not sure it would provide a lot of advantage, as the users are already used to doing it by themselves right now.
Does anyone have ideas ?
---
### Author:  [@worldofpeace](https://discourse.nixos.org/u/worldofpeace)
#### Post date: [October 6, 2019, 5:01pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/7 "2019-10-06T17:01:37Z")
> [@DianaOlympos](#):
>
> In the meantime, we could offer something like Go, and use the non sandbox classic “getting deps” of the tooling in the `build` tools. I am not sure it would provide a lot of advantage, as the users are already used to doing it by themselves right now.
Do you mean `buildGoModule`? Where one derivation is getting the dependencies and another for the build?
> details on how that deriver works [https://kalbas.it/2019/03/17/announcing-the-new-golang-infrastructure-buildgomodule/](https://kalbas.it/2019/03/17/announcing-the-new-golang-infrastructure-buildgomodule/)
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [October 6, 2019, 5:57pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/8 "2019-10-06T17:57:28Z")
Yes it is the one i had in mind. Thank you for the link, i learned something today with the `outputHash` stuff . I will explore tomorrow or later today a bit more. But yes, it is the one i am thinking of, where one derivation get the dependencies and another the build.
If you have any other idea, i take it obviously 🙂
The bigger “problem” is that we are kind forced to write an erlang program to read the lockfile, as it is an ETS dump. At least if we want to make it nice.
I do not know if it would be a nice step forward, as you still need to generate the intermediary hash of the package derivation, so we will have to document and explain that too 🙂
---
### Author:  [@unthought](https://discourse.nixos.org/u/unthought)
#### Post date: [May 5, 2020, 12:22pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/9 "2020-05-05T12:22:00Z")
Resurrecting this, as it seems to be the most recent discussion concerning the BEAM Ecosystem (incl. Elixir).
Just wanted to raise another concern: at least Elixir supports compile time configuration of dependencies, so package build output can differ even from the same origin, depending on what’s in config.exs, etc. of the application project. So sources can only reliably be built as dependency of an actual application, and not in isolation.
One well-known example is [Logger](https://hexdocs.pm/logger/Logger.html#module-application-configuration). It supports compile time elimination of log statements (you can completely remove all debug output at compile time). Since it’s part of Elixir itself, it potentially also impacts libraries written in Elixir making use of Logger (in this case, since it’s an optimization, you could just set the runtime configuration correctly, and you just lose performance, but there are other examples where configuration is baked into the compiled code).
I’m unsure how common this usecase is, but it’s a big drag for having prebuilt derivations in nixpkgs (unless trying to heuristically build for common configuration sets, or figure out which libraries are pure in this regard (not easy)).
However, good support for build derivations from mix projects (essentially mix.exs tree and mix.lock) would still be tremendously useful for the superior caching that Nix can provide, as long as the derivations include all inputs. To make use of improved caching, at least the deps have to be in their own derivation as Go seems to be doing with the fetcher.
Has there been any recent activity on the BEAM Ecosystem in that direction?
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [May 5, 2020, 12:44pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/10 "2020-05-05T12:44:41Z")
Hi!
Yes indeed, the config need to be in there too and that was part of my goals.
Outside of that, no i don’t think more work has been done. I have not time right now but i need it…
---
### Author:  [@samrose](https://discourse.nixos.org/u/samrose)
#### Post date: [May 23, 2020, 2:10am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/11 "2020-05-23T02:10:05Z")
What would the problem be with just using or expanding [GitHub - transumption/mix-to-nix: Sandboxable mix.lock and rebar.lock Nix generator](https://github.com/transumption/mix-to-nix) ?
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [June 22, 2020, 10:43am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/12 "2020-06-22T10:43:18Z")
Sorry for the late reply, was a bit busy.
Basically, this try to read the mix.lock and translate it into a nix derivation. The problem is that it means we depends on the mix.lock format, which afaik is not considered stable.
Also, in general, i do not consider trying to replicate the work the package management of the language ecosystem do inside nix to be the best solution. We are seeing the pain it is generating for haskell or python. I would far prefer letting them do their job and making sure that the output is safe through hashing. I understand why it is not preferred either by a part of the nixpkgs community.
At this point, i would far prefer something that use mix/rebar3/hex\_core to handle the download, which they do in a fair known directory. And then validate that the content of said directory is stable, as `buildGoModules` does. See [Nixpkgs 23.11 manual | Nix & NixOS](https://nixos.org/nixpkgs/manual/#ssec-go-modules)
I put it on hold due to some quirks of rebar3 and how mix handled it, which made this a little bit more painful than it should. And simply no time so far to do it. But there is work in progress to make this nicer and i do hope to finally get time to do this in the next month or so.
---
### Author:  [@samrose](https://discourse.nixos.org/u/samrose)
#### Post date: [September 27, 2020, 11:23pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/13 "2020-09-27T23:23:12Z")
So, actually what @anon31783435 delineates in the example above is likely the best approach currently for elixir projects.
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [September 29, 2020, 9:43am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/14 "2020-09-29T09:43:53Z")
Yes.
It is something that the nix core team is against and that Flakes in particular try to reduce, but it does not cover our needs at all for now.
So yes, for now, the best approach is, sadly, to disable the sandbox and use @anon31783435 method.
---
### Author:  [@anon31783435](https://discourse.nixos.org/u/anon31783435)
#### Post date: [October 1, 2020, 2:32pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/15 "2020-10-01T14:32:48Z")
Another (probably better) approach is the one done by @hauleth - I haven’t explored in detail yet.
→ [github.com/hauleth/nix-elixir](https://github.com/hauleth/nix-elixir)
Mentioned over here:
- [What's the current best-in-class approach to packaging Elixir/Erlang/BEAM applications using Nix+releases as of July 2020? - #6 by anon31783435](https://discourse.nixos.org/t/whats-the-current-best-in-class-approach-to-packaging-elixir-erlang-beam-applications-using-nix-releases-as-of-july-2020/8205/6)
- [github.com/NixOS/nixpkgs/pull/97245](https://github.com/NixOS/nixpkgs/pull/97245)
On my side, with the compilation time improvements within Elixir v1.11,
I will start to get even more “un-pure” and try to reuse `_build` & `deps`.
Build time within CI is a major important thing for me.
But I’ve already mentioned I use Nix as a sharable development environment setup
and for perfect fast cache-able CI.
So my preferences may differ.
* * *
Other things about the state of the BEAM within Nix:
I’ve prepared the update of Erlang to v23.1 → [github.com/NixOS/nixpkgs/pull/98646](https://github.com/NixOS/nixpkgs/pull/98646).
But it’s once again slow in getting adopted.
For me to have a perfect development environment it’s important to have latest versions fast available…
I also would prefer if we don’t prolong the update of the default `Erlang` within Nix too long.
So in my opinion it should be time now to change from v22 to v23.1.
I’ve a look into upgrading `rebar3` to the latest version, as the current one would probably not build with Erlang v23.
But the `rebar3` pkg seems too complicated for my Nix skills…
Good for now - all the best.
---
### Author:  [@hauleth](https://discourse.nixos.org/u/hauleth)
#### Post date: [October 1, 2020, 2:54pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/16 "2020-10-01T14:54:00Z")
Well, in the OpenTelemetry I needed Rebar 3.4.1:
```auto
{ pkgs ? import {} }:
with pkgs.beam.packages.erlang;
let
rebar = rebar3.overrideAttrs (oldAttrs: {
version = "3.14.1";
src = pkgs.fetchFromGitHub {
owner = "erlang";
repo = "rebar3";
rev = "3.14.1";
sha256 = "113yy3scj6dxf0mr1y7jivrh4gdcyhp2wr91j9gc53v9q2jxc8ly";
};
});
in
pkgs.mkShell {
buildInputs = [erlang rebar elixir];
MIX_REBAR3 = rebar;
}
```
So it isn’t that hard.
---
### Author:  [@anon31783435](https://discourse.nixos.org/u/anon31783435)
#### Post date: [October 1, 2020, 6:23pm UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/17 "2020-10-01T18:23:17Z")
So for bringing an up-to-date `rebar3` version upstream to nixpkgs I only need to address `src`?
→ [rebar3 - default.nix](https://github.com/NixOS/nixpkgs/blob/cc9b34a56bd1af407568e9dd6b7efbc93ff536c1/pkgs/development/tools/build-managers/rebar3/default.nix)
The various `fetchHex` thinks can be ignored?
---
### Author:  [@berbiche](https://discourse.nixos.org/u/berbiche)
#### Post date: [October 2, 2020, 2:22am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/18 "2020-10-02T02:22:54Z")
You only need to update the `fetchHex` if the `rebar.lock` file has changed
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [October 19, 2020, 10:11am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/19 "2020-10-19T10:11:35Z")
I agree on moving to 23.1 asap.
I have been working on cross compilation [Erlang: fix cross compilation by DianaOlympos · Pull Request #100835 · NixOS/nixpkgs · GitHub](https://github.com/NixOS/nixpkgs/pull/100835)
This is not perfect, mostly due to how it handles bootstrapping the BEAM. It needs the same version of OTP than you want to build and right now i inject the default `erlang_nox`.
I am wondering if i should not do an import from derivation with a specific bootstrap builder, but that means building a specific bootstrapped version everytime we cross compile… I am not against that but i have no idea what the nixpkgs maintainer would think of it.
I also agree on the reactivity problem. Afaict we do not have anyone with the merge right that is actively working on BEAM stuff, which limits a bit our speed of getting things merged
---
### Author:  [@DianaOlympos](https://discourse.nixos.org/u/DianaOlympos)
#### Post date: [October 19, 2020, 10:21am UTC](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202/20 "2020-10-19T10:21:17Z")
I also begun moving Riak to 3.0.1 so we can drop R16-basho, but they now use rebar3 without a lockfile, making the whole thing a bit of a pita. I can probably do it, but it is lower down the list. If someone else want to look at it, i already have most of the change ready, so just ask and i can try to help you ?
[Next page](https://discourse.nixos.org/t/state-of-the-beam-ecosystem-in-nix/4202.md?page=2)