Adding rescript package to nodePackages locally

Help
1

Hello! I’m trying to get rescript, which isn’t included in nixpkgs, up and running locally. What I’ve tried so far is adding the following to my configuration.nix:

nixpkgs.overlays = [
    (self: super: {
      rescript = super.nodePackages.rescript.override {
          src = super.fetchFromGitHub {
              owner = "rescript-lang";
              repo = "rescript-compiler";
              rev = "e0921fa9e67bdf5c7ee301599689db1d9b9894b6";
              sha256 = "07vkzdq7qwy5djm37jaxld5m9d55d5aa6bw5s286gm4r139102q4";
              };
      };
    })
  ];

So running sudo nixos-rebuild switch doesn’t break, which is nice, but it also doesn’t appear to add rescript to my environment (i.e.- I can’t run it from bash). I’m not really sure why this is. Then it occurred to me that what I really want is to add it to my project’s flake and have it available as one of the devShell.buildInputs, but where exactly should it go to make it available here?

I’ve very likely taken a wrong turn since I’m still very new to nix, so I’d be grateful for any corrections to my bad assumptions! In summary my questions are:

  • What have I done wrong in adding the overlay to my configuration.nix?
  • Where does an additional nixpkgs package go in a flake.nix? Under the outputs.overlays? Somewhere else?

I’m not sure my flake.nix brings too much context, but here it is for reference:


  description = "Project rescript bindings";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    f-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, f-utils }:
    let
      name = "re-proj";
    in
    f-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.x86_64-linux;
      in {
        devShell = pkgs.mkShell {
          name = "proj-shell";
          buildInputs = with pkgs; with pkgs.nodePackages; [
            # dev packages
            nodejs-16_x
            bs-platform
            node2nix
            yarn
            rescript # This breaks things since it doesnt' refer to anything

            python39
          ];
          shellHook = ''export PATH="./node_modules/.bin:$PATH"'';
        };
      });
  ];
}
2

The overlay is set up correctly, but you’re misunderstanding how overlays work. Overlays don’t magically create or install packages; that you still need to do yourself.

An overlay does exactly what it says on the tin - it “overlays” some stuff on top of the normal nixpkgs contents. I.e., by setting some attribute (read dictionary key in JavaScript lingo) in the overlay’s attrset (dictionary), nix will put the contents of what you specify in the nixpkgs used for your build later on.

In other words, you’ve created a rescript attribute, and if you use pkgs.rescript anywhere in your configuration it will refer to the value you set there.

What you also fail to realize is that nix is lazy. Just because you created that attribute doesn’t mean nix will use it, or even build whatever you define. You actually need to use the attribute somewhere where nix is forced to do something with it, like in environment.systemPackages. So, if you set:

environment.systemPackages = with pkgs; [ rescript ];

Anywhere in your config, that will actually refer to the package you tried to create there, and try to include it in your system path.

Unfortunately, this won’t work. The value of your attribute is going to try and evaluate, find that the original nixpkgs didn’t contain a rescript, and therefore there is no super.rescript whose override function you could use.

You’ll need to write an actual nodejs package instead of trying to override something. This, unfortunately, isn’t trivial, because npm is npm and a massive pain to work with due to thousands of nested dependencies. The documentation for this is in this section of the nixpkgs manual, but it’s quite the topic: Nixpkgs Reference Manual

To include it in a devShell, the easiest option for overriding packages is simply to import the flake, and set the overlays option:

    let
        overlays = [<your overlays>];
        pkgs = import nixpkgs { inherit overlays; };
      in {
        devShell = pkgs.mkShell {
          name = "proj-shell";
          buildInputs = with pkgs; with pkgs.nodePackages; [
            # dev packages
            nodejs-16_x
            bs-platform
            node2nix
            yarn
            rescript # If set in your overlays, now works

            python39
          ];
          shellHook = ''export PATH="./node_modules/.bin:$PATH"'';
        };
      });

This works because the flake will resolve to the actual directory of the nixpkgs you refer to in the inputs, and therefore import-ing it will import the normal nixpkgs default.nix, which in turn just calls the function from the file I link to earlier. You can also set any of the other arguments of that function - config is especially useful, and can be used to allow non-free applications and such for nixpkgs that aren’t configured with your system configuration.

Note that this is slower than your version, because import-ing nixpkgs actually requires resolving a bunch of things, whereas referring directly to the legacyPackages does not - when you need to overlay there is no way around that, though.

That said, if you want to create a new package, it’s probably better to do something like this:

     let
        pkgs = nixpkgs.legacyPackages.x86_64-linux;
      in {
        devShell = pkgs.mkShell {
          name = "proj-shell";
          buildInputs = with pkgs; with pkgs.nodePackages; [
            # dev packages
            nodejs-16_x
            bs-platform
            node2nix
            yarn
            (callPackage ./rescript.nix { })

            python39
          ];
          shellHook = ''export PATH="./node_modules/.bin:$PATH"'';
        };
      });

The callPackage there is a useful helper that just takes a function and gives it all the arguments it asks for from pkgs, and is generally used to create packages. Read the nix pills to learn how it works :slight_smile: My intention there is to say “do whatever you would to create a package”, which might not be obvious if you don’t know how to do that yet. I’d recommend reading this: https://nixos.org/manual/nix/stable/expressions/simple-expression.html and this too: Nixpkgs Reference Manual

I realize this is all a lot to take in, but I hope it helps - ask away if you need more help! On the up side, once you grok all this there’s not much else to learn about nix.

Overall, I’d suggest learning how to do this with something more straightforward than npm packages. It’s not the easiest thing to understand, and the complexities of thousands of dependencies will just make it harder.

Also, maybe take it one step at a time; understand how to package something with nix, then understand how to include that in your devShell, and then understand how to include packages in your overlay. After that you can learn about override (and more practically, overrideAttrs), which is how you make small changes to existing packages, and the typical use case for overlays.

4 Likes
3

First off, I really appreciate the well-thought-out response and the code examples. Thank you for taking the time to break it down for me!

Overlays don’t magically create or install packages; that you still need to do yourself.

Yep! I realized I was on the wrong track a little while after posting, that said I didn’t realize how much work it would actually take to get rescript up and running!

What you also fail to realize is that nix is lazy. Just because you created that attribute doesn’t mean nix will use it, or even build whatever you define.

Correct! I’ve read it, I just need to remember it. :slight_smile:

I realize this is all a lot to take in, but I hope it helps - ask away if you need more help! On the up side, once you grok all this there’s not much else to learn about nix.

Overall, I’d suggest learning how to do this with something more straightforward than npm packages. It’s not the easiest thing to understand, and the complexities of thousands of dependencies will just make it harder.

That’s finally starting to sink in for me. Since I’ve already committed to NixOS, it looks like the best course of action is going to be falling back to buildFHSUserEnv and chipping away at a “pure” nix experience in parallel in such a way that it doesn’t block dev activities.

This post gives me a lot to draw from and should keep me busy for a while. I’ve been hedging on actually reading the Nixpkgs manual, but it sounds like this, along with the other links you’ve provided, will be the next step(s)!

One more question for you or anyone who has a good answer. Are there any secrets for debugging and code completion in nix? Or is the answer some combination of “use the repl”, “RTFM”, search GH, etc.? For instance, you mentioned above using legacyPackages, which I went to look up in the Nixpkgs manual but couldn’t find, but did manage to find this thread here in the forum. You also linked to the overlays option, which is super useful to look over, but would be a needle-in-a-haystack for me to find had you not linked it! Honestly, even if the answer is something like: Ctrl-F in the Nixpkgs manual followed by a GH search of usage, that would at least let me know I’m not missing out on a better approach!

4

There are definitely some battles you can’t win. I’ve given up on developing Bazel on NixOS directly, for example, my trick for this has become developing some things in a virtual machine with my usual desktop environment/emacs/other things installed through home-manager. gnome.gnome-boxes gives a very comfortable experience for that.

I’ve found projects using mostly pure variants of either JS/Python/Rust/C/++, and some more niche langs, to work quite well once you understand both the quirks of nix and the packaging/build systems of those languages, though. It’s mostly specific projects that depend too much on their hosts being ubuntu, or like mixing languages in particularly horrible ways, that are hard to develop on.

It’s still significantly better than having random applications installed on my host and worrying about the next time I have to reinstall a computer (or just breaking things by installing a more modern python). Hopefully one day everybody will be using nix and noone will be writing terrible packages that install binaries at run time anymore.

The latter, and hang out on this discourse :wink: Flakes especially are a bit under-documented. My go-to are the nixpkgs and nix manuals, the nixos manual when I need something modules related and the nixpkgs source when the former three failed. Flakes didn’t really click for me until I’d spent a day or so figuring out the module code in nixpkgs. https://nix.dev is also very worth a read, as is Xe’s blog: https://christine.website/blog/series. Also you should know about https://sourcegraph.com/search if you don’t like using grep locally, because github’s search functions are terrible.

This is probably the biggest weakness of the nix ecosystem, but it’s less bad than most people seem to think. RTFM very much applies, the nixpkgs and nix manuals are gold troves too many people skip (probably because they’re terse and long, @ryantm is working on a better theme for the nixpkgs one here), and the nixpkgs source is far nicer to read than most source repositories - and has an excellent git history, too!

That doesn’t mean there isn’t a documentation problem :wink:

There is a language server for some IDE integration, though: GitHub - nix-community/rnix-lsp: WIP Language Server for Nix! [maintainer=@aaronjanse], and also a debugging lib in nixpkgs: lib.debug: debugging functions | nixpkgs. There’s also alejandra if you were looking for autoformatting. Given the nix language isn’t strict about typing, doesn’t have doc comments and also has a relatively small community, they aren’t quite as good as modern debuggers/lsps/etc for say, Rust (but still pretty damn good, considering).

2 Likes
5

This is great stuff and leaves me in much better shape! I took pretty much all of your advice here even if it’s not clear on discourse (I right-click links to open them). Alejandra and Sourcegraph were two particularly big wins for me and the reading links have all been either pinned or bookmarked. :pray:

Great to know slogging through the manuals, monitoring the forum and reading the source is more-or-less the right track. Also good to know that experienced Nix folks are also carefully picking their battles when developing SW and considering nixifying their repos!

Again, thanks for the detailed responses. It gives me a lot to review and really irons out my path forward. It really is a huge help!

1 Like