What is the best way to use Serverless Framework on NixOS?

Help
1

Wanted to try out Serverless via nix-shell so I looked it up, ran the command

nix-shell -v -p nodePackages.serverless

but it is hopelessly outdated, and it also doesn’t work:

$ serverless
#...
  Error --------------------------------------------------

  Error: EACCES: permission denied, open '/home/toraritte/shed/flox-aws-serverless/my-project/serverless.yml'
      at Object.openSync (fs.js:462:3)
      at Object.writeFileSync (fs.js:1384:35)
      at renameYmlService (/nix/store/p8xhrzb9y62wrl3mgd517hgmfrbi6wl4-node_serverless-1.81.1/lib/node_modules/serverless/lib/utils/renameService.js:20:7)
      at renameService (/nix/store/p8xhrzb9y62wrl3mgd517hgmfrbi6wl4-node_serverless-1.81.1/lib/node_modules/serverless/lib/utils/renameService.js:44:5)
      at /nix/store/p8xhrzb9y62wrl3mgd517hgmfrbi6wl4-node_serverless-1.81.1/lib/node_modules/serverless/lib/utils/createFromTemplate.js:60:15
  From previous event:
      at /nix/store/p8xhrzb9y62wrl3mgd517hgmfrbi6wl4-node_serverless-1.81.1/lib/node_modules/serverless/lib/utils/createFromTemplate.js:59:26
      at /nix/store/p8xhrzb9y62wrl3mgd517hgmfrbi6wl4-node_serverless-1.81.1/lib/node_modules/serverless/node_modules/graceful-fs/polyfills.js:243:20
      at FSReqCallback.oncomplete (fs.js:156:23)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          linux
     Node Version:              12.21.0
     Framework Version:         1.81.1
     Plugin Version:            3.8.2
     SDK Version:               2.3.1
     Components Version:        2.34.9

Found this reddit post about this exact same issue, but I’m having trouble understanding the recommendations of this comment (quoted at the bottom), so I used this approach for now, but it is far from ideal:

# https://github.com/NixOS/nixpkgs/issues/3393#issuecomment-50330167
$ npm config set prefix ~/mutable_node_modules

$ nix-shell -p nodejs

[nix-shell:~/my/work/dir]$ npm install -g serverless

[nix-shell:~/my/work/dir]$ tree -L 2 ~/mutable_node_modules/
├── bin  
│   ├── serverless -> ../lib/node\_modules/serverless/bin/serverless.js  
│   └── sls -> ../lib/node\_modules/serverless/bin/serverless.js  
└── lib  
└── node\_modules  

[nix-shell:~/my/work/dir]$ ~/mutable_node_modules/bin/serverless --version
Framework Core: 2.62.0
Plugin: 5.4.6
SDK: 4.3.0
Components: 3.17.1
...

For completeness sake, here’s the post mentioned above:

CSI_Tech_Dept·5m·edited 5m

I never used the provided serverless because it is updated very frequently and then there’s also plugins which I’m sure you’ll run into once this is resolved.

What I previously did was entering a shell with nodejs, and then using npm to install it, and npx to execute.

Basically when I have project.json and project-lock.json files, I just run npm ci .

After serverless merged PR that made it search for plugins in its own package directory things were simplified.

I’m using GitHub - nix-community/npmlock2nix: nixify npm based packages [maintainer=@andir] and something like this:

    serverless_project = pkgs.runCommandLocal "serverless-env" {} ''
      mkdir $out
      cp ${./package.json} $out/package.json
      cp ${./package-lock.json} $out/package-lock.json
    '';

    node_modules = npmlock2nix.node_modules {
      src = serverless_project;
      nativeBuildInputs = [
        pkgs.bash
        pkgs.python38
      ]
      ++ lib.optional pkgs.stdenv.hostPlatform.isDarwin pkgs.darwin.cctools;
    };

Then node_modules can be then included in mkShell for example (this should make sls command available in it). That makes Nix build the serverless and plugins (based on the lock file). The reason in src I’m not providing current dir, like ./. and instead creating the serverless_project derivation is because I like to keep these files together with my project and not make Nix rebuild whole thing when smallest change is made. Building nodejs in Nix is long process, because node has tons of tiny packages, something that Nix isn’t optimized to handle well.

… and the gentleperson was kind enough to provide the explicit steps as well:

CSI_Tech_Dept - 8m:

Here is an example of a full setup, of course you can customize it. For example instead of using niv you can use fetchFromGitHub etc. I use niv because I want things to be pinned, so I can get a reproducibility. I think flakes will replace that but I have not get around trying it yet.

Anyway, here is how to set it up:

nix-shell -p niv nodejs
niv init -b nixpkgs-unstable
niv add tweag/npmlock2nix
echo '{"name": "serverless-project", "version": "0.1.0", "private": true}' > package.json
npm install --save-dev serverless serverless-step-functions
rm -rf node_modules # kind of important as serverless will still use this directory if it exists
exit

Create file shell.nix like this:

let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {
    overlays = [
      (self: super: {
        npmlock2nix = pkgs.callPackage sources.npmlock2nix {};
      })
    ];
  };

  # This ensures that serverless won't be rebuild every time you change
  # an unrelated file
  serverless_project = pkgs.runCommandLocal "serverless-env" {} ''
    mkdir $out
    cp ${./package.json} $out/package.json
    cp ${./package-lock.json} $out/package-lock.json
  '';
  node_modules = pkgs.npmlock2nix.node_modules {
    src = serverless_project;
    nativeBuildInputs = with pkgs; [
      bash
      python39
    ]
    ++ pkgs.lib.optional pkgs.stdenv.hostPlatform.isDarwin pkgs.darwin.cctools;
  };
in pkgs.mkShell {
  packages = [
    pkgs.niv
    pkgs.nodejs
    node_modules
  ];
}

Then execute:

nix-shell # this will take a while at first run
sls --version

I highly recommend installing and configuring direnv for example nix-env -iA nixpkgs.direnv and configure it with one of integrations that caches. For exmaple nix-direnv (you can install it by invoking nix-env -iA nixpkgs.nix-direnv (don’t forget that you still need to configure it).

Then you can type:

direnv edit .

and place:

use nix

It will allow you to get your dev env by just entering the directory. If you need it to be refreshed do touch shell.nix .

1 Like
2

The next step will be to figure out this one:

takeda 6 months ago [[–]](javascript:void(0))

So nix is kind of weird thing, because parts of it you can use to replace part of components but hen you can create a whole solution (and ultimately that works better). In that case it’s like asking to write a tutorial for python.

To illustrate what I mean here is my history with nix (simplified, but it’s meant to show the evolution):

So I started using nix as a reproductive dev environment. Initially using shell.nix to have all my tooling. This is great if you also have direnv installed, which basically automatically makes tools related to the project to appear.

Anyway I started then using various tools that would translate python project to nix initially was also a big fan of setup.py/cfg and pip-tools. I wasn’t happy with either of them until I found poetry2nix. Poetry is also quite decent so now that’s my favorite way to build a project.

So what poetry2nix does is translates on the fly pyproject.toml and poetry.lock file to nix expression. With it you can build project in nix you can also create a shell that has python with all packages available as described by poetry. So that’s now quite decent reproducible environment.

I realized that to make the dev environment nice I’m adding a lot of nix boilerplate to every project, also as I’m learning new things and improving it it is getting hard to synchronize that boilerplate across other projects.

So I created a new repo where I put everything there, I also learned how to use modules system (which they created for NixOS) ultimately I made repo that you reference in default.nix and shell.nix in my project it loads project.nix file which describes information about my project. Allows me to set various aspect of the application, like what additional packages should be installed in dev shell (maybe I need a local postgresql installation). How the project will be deployed (if it is meant to run as AWS lambda, a serverless (npm application) is added and package.json/package-lock.json files are sourced. If it is a docker asks what binary should be executed in the container. Whether to compile using minified python (with missing some of core libraries) etc.

That ended to be some Nix code, but simplified things greatly, and makes things usable by people not familiar with Nix.

We use gitlab for CI/CD so I created a special gitlab-runner for nix builds. It actually is already built in in NixOS and only needs to be enabled, and they also show configuration in example. Basically how it works is that it stills spins a docker container for the build, but exposes /nix/store (read only) and the unix socket to trigger build. I also created an S3 bucket and configured it for storing caches of builds.

Having this runner lets me utilize one of major benefits of Nix which is caching (if all inputs (source files, architecture, interpreters etc) don’t change then output will be the same). This is great, because if I only change source file, and don’t change dependencies, the pipeline will only rebuild my app. If I work on a branch and branch passes all tests, I merge it and merge is just fast forward, then build job in master branch just takes couple seconds, because nothing change. If the branch wasn’t rebased and actual merge was done and some files are now different the nix will rebuild things that were modified.

Deployment of my python app as a lambda ends up being just:

    nix-shell --run "sls deploy"

nix-shell takes care that the dev environment is created and inside of it invokes “sls deploy” command. This similarly utilizes caching (that persists between builds) so it is faster than what we normally used like loading a docker container and then running it.

This is quite decent and I’m still testing it on a single app, but it basically cut deployment time from 10 minutes to 5 minutes. And I know I can cut it further (serverless doesn’t know about nix, but it understands poetry, so once again again it downloads all dependencies it needs, extracts them and bundles them into the lambda. I plan to populate its cache from the packages that nix stores when is running the build).

There is also possibility that I’m aware of, but haven’t explored yet, but seems interesting. It might require some work. For example you could configure your local computer to use the machine that is used for CI/CD build to help with local builds. Similarly you could expose the cache as well to prevent unnecessary rebuilds. You probably shouldn’t do that if the user isn’t trustworthy, otherwise you might need to create some additional safety layer.

Now in your scenario you have bare bone servers to deploy. If those would run NixOS I believe the deployment step would be simply to deliver new configuration.nix referencing your application to them and also have those machines configured to use the same cache as you had with CI/CD so they won’t have to rebuild everything again. There are several tools that make the process easier, for people who use public cloud there’s terraform support, NixOps (last time I used it was the weakest part of Nix, but maybe it improved, it was great tool when I wanted to test everything a la hashicorp’s vagrant)

1 Like