State of the BEAM Ecosystem in Nix

Development
10

Hi!

Yes indeed, the config need to be in there too and that was part of my goals.

Outside of that, no i don’t think more work has been done. I have not time right now but i need it…

1 Like
11

What would the problem be with just using or expanding https://github.com/transumption/mix-to-nix ?

1 Like
12

Sorry for the late reply, was a bit busy.

Basically, this try to read the mix.lock and translate it into a nix derivation. The problem is that it means we depends on the mix.lock format, which afaik is not considered stable.

Also, in general, i do not consider trying to replicate the work the package management of the language ecosystem do inside nix to be the best solution. We are seeing the pain it is generating for haskell or python. I would far prefer letting them do their job and making sure that the output is safe through hashing. I understand why it is not preferred either by a part of the nixpkgs community.

At this point, i would far prefer something that use mix/rebar3/hex_core to handle the download, which they do in a fair known directory. And then validate that the content of said directory is stable, as buildGoModules does. See NixOS - Nixpkgs 21.05 manual

I put it on hold due to some quirks of rebar3 and how mix handled it, which made this a little bit more painful than it should. And simply no time so far to do it. But there is work in progress to make this nicer and i do hope to finally get time to do this in the next month or so.

6 Likes
13

So, actually what @anon31783435 delineates in the example above is likely the best approach currently for elixir projects.

1 Like
14

Yes.

It is something that the nix core team is against and that Flakes in particular try to reduce, but it does not cover our needs at all for now.

So yes, for now, the best approach is, sadly, to disable the sandbox and use @anon31783435 method.

2 Likes
15

Another (probably better) approach is the one done by @hauleth - I haven’t explored in detail yet.
→ github.com/hauleth/nix-elixir

Mentioned over here:

On my side, with the compilation time improvements within Elixir v1.11,
I will start to get even more “un-pure” and try to reuse _build & deps.
Build time within CI is a major important thing for me.

But I’ve already mentioned I use Nix as a sharable development environment setup
and for perfect fast cache-able CI.
So my preferences may differ.

Other things about the state of the BEAM within Nix:

I’ve prepared the update of Erlang to v23.1 → github.com/NixOS/nixpkgs/pull/98646.
But it’s once again slow in getting adopted.
For me to have a perfect development environment it’s important to have latest versions fast available…

I also would prefer if we don’t prolong the update of the default Erlang within Nix too long.
So in my opinion it should be time now to change from v22 to v23.1.

I’ve a look into upgrading rebar3 to the latest version, as the current one would probably not build with Erlang v23.
But the rebar3 pkg seems too complicated for my Nix skills…

Good for now - all the best.

2 Likes
16

Well, in the OpenTelemetry I needed Rebar 3.4.1:

{ pkgs ? import <nixpkgs> {} }:

with pkgs.beam.packages.erlang;

let
  rebar = rebar3.overrideAttrs (oldAttrs: {
    version = "3.14.1";
    src = pkgs.fetchFromGitHub {
      owner = "erlang";
      repo = "rebar3";
      rev = "3.14.1";
      sha256 = "113yy3scj6dxf0mr1y7jivrh4gdcyhp2wr91j9gc53v9q2jxc8ly";
    };
  });
in
  pkgs.mkShell {
    buildInputs = [ erlang rebar elixir ];

    MIX_REBAR3 = rebar;
  }

So it isn’t that hard.

4 Likes
17

So for bringing an up-to-date rebar3 version upstream to nixpkgs I only need to address src?
→ rebar3 - default.nix

The various fetchHex thinks can be ignored?

1 Like
18

You only need to update the fetchHex if the rebar.lock file has changed

1 Like
19

I agree on moving to 23.1 asap.

I have been working on cross compilation Erlang: fix cross compilation by DianaOlympos · Pull Request #100835 · NixOS/nixpkgs · GitHub
This is not perfect, mostly due to how it handles bootstrapping the BEAM. It needs the same version of OTP than you want to build and right now i inject the default erlang_nox.
I am wondering if i should not do an import from derivation with a specific bootstrap builder, but that means building a specific bootstrapped version everytime we cross compile… I am not against that but i have no idea what the nixpkgs maintainer would think of it.

I also agree on the reactivity problem. Afaict we do not have anyone with the merge right that is actively working on BEAM stuff, which limits a bit our speed of getting things merged

1 Like
20

I also begun moving Riak to 3.0.1 so we can drop R16-basho, but they now use rebar3 without a lockfile, making the whole thing a bit of a pita. I can probably do it, but it is lower down the list. If someone else want to look at it, i already have most of the change ready, so just ask and i can try to help you ?

1 Like
21

Nice, thanks @hauleth and @anon31783435 for this. But… but fetchMixDeps still has to be run in non-sandboxed mode, am I right or I’m missing some key things?

1 Like
22

You are missing to provide the sha256 in the attribute set to make fetchMixDeps create a FOD.

1 Like
23

FOD? Please explain yourself thinking that I’m totally dumb

1 Like
24

Fixed Output Derivation.

A derivation of which you tell nix in advance that it has “side effects” but promise that those side effects will lead to content that has a certain hash.

You have to pass this hash to to fetchMixDeps.

You can use lib.fakeSha256 as an initial value and then copy the got from the error message you see during build.

This is called “trust on first use”.

3 Likes
25

Small Update.

  • No more hermetic rebar3
  • No more snapshot/package set
  • updated the documentation
  • Got rid of Riak-CS and Stanchion. Not maintained anymore. One less dep on R16Basho

Now on the new problems

  • Still no decision forward on how to handle mix/rebar lockfiles. I am still in favour of a FOD tool but well still problematic
  • Riak-3.0.1 have shown that problem even better, they moved to rebar3 which means that now i have to find a nice way to bring the packages and the whole dependencies properly. I have not done the work so far which means
  • R16-Basho still live. But in progress
  • The whole beampackages library and organisation is still a mess, that should be able to clean up better
  • Some work on cross compiling has been done, but this is stale due to bootstrapping problems. Still searching for someone that understand how the splicing is done to help there
  • The doubling of erl/libs in the path is still there, noone has taken the time to test things there.
  • It seems that buildMix is broken now ?
3 Likes
26

Hey BEAMers.

I’ll use this thread for a question.
Should we really include all of the docs in the Erlang build (HTML, PDF, chunks)?
Wouldn’t it be enough to include the chunks only for most of our usecases?
(Note: install-docs / environment variable DOC_TARGETS=chunks)

I encountered this while in need to deliver a Docker image to one of my clients.
The image is much larger then it should be, caused by the massive amount of docs.
I don’t understand why Erlang lands in the image anyway.
Theoretically the included ERTS of my mix release should already be enough.?

Thanks everybody improving BEAMland in Nixpkgs.
Thanks for erlang_nox / beam_nox (we still need to mention this better in the docs).

Thanks DianaOlympos for your effort in cross compiling.
I still hope someone with experience will help out getting this foreward.
I still find Nix will give the best expierince for cross compiling Erlang…

1 Like
27

Depending on how you built the docker image, especially when doing an embedded release, there might be a reference to the original erlang used at build time. This was at least the case when building releases for some elixir application I started in a toy project.

In that project I used the following in the fixupPhase to mitigate the problem.

for f in $(${pkgs.findutils}/bin/find $out -name start); do
  substituteInPlace $f \
    --replace 'ROOTDIR=${packages.erlang}/lib/erlang' 'ROOTDIR=""'
done

On top of that I was able to further reduce image sizes by building erlang without wx and systemd.

3 Likes
28

Thanks NobbZ. Your snippet seems to be the right pice.
Finally building Erlang happens nowadays in an acceptable amount of time (10 minutes).
Former I needed to wait an hour - never found out why.

But I ran into another trouble.
Building on my Alpine (musl) a wrong OpenSSL path get’s linked in my build crypto lib (lib/crypto/musl_xxx.so).
I don’t know yet how to patch this…

My question to all Erlang maintainers or BEAM team members is still open.
Should we really include all docs (HTML, PDF) in the Erlang builds?

In my oppinion the chunks could be enough!?
I could fill a PR if anybody is in the same oppinion.
(finally as building happens fast enough :wink:

29

Linking the Status of lang2nix approaches discussion here because it seems pertinent that there is still no accepted approach (or the conversation continued elsewhere, and I totally missed it).