Use nix in netlify

nomeata · February 14, 2022, 5:43pm

I am exploring ways to host some of my mostly static websites not on my own server, but “in the serverless cloud”, and am looking at netlify. I am using some small haskell scripts in my build process, and surely enough, their default build image does not have haskell installed.

For reasons I don’t have to explain here I would prefer to use nix to get the dependencies I need. Official nix support in netlify seems to be stalled , and I did not find any hints as to how to work the lack of official support, which surprised me.

Does anyone here use nix as part of a netlify build step?

nomeata · February 14, 2022, 6:27pm

I guess I could use Github Actions to render the files and push to a branch (or even separate repo), and then host that on netlify. But I’d lose some of the nice features of netlify, e.g. previews for PRs…

roberth · February 14, 2022, 8:51pm

I’ve started using Hercules CI Effects to deploy to netlify since recently.
Here’s how: https://github.com/hercules-ci/blog.hercules-ci.com/pull/61/files
It does use the experimental agent 0.9 configuration format, but the effect itself would be the same in 0.8 (stable).
0.9 got delayed so I have yet to do the recursive thing that is blogging about the blog.

nomeata · February 14, 2022, 9:03pm

That link is 404 here. Private repo?

roberth · February 14, 2022, 9:14pm

Oops, thought it was already public, like docs.hercules-ci.com

nomeata · February 14, 2022, 9:22pm

Thanks! I see that you pass the branch name to netlify as you push the result, but nothing more. Isn’t there a risk of a race condition? If I push to the repo twice in short succession, and the first build finishes after the second, will the wrong one be live?

jtojnar · February 15, 2022, 10:45am

nixos.org seems to use GitHub actions and there are PR previews:

github.com

NixOS/nixos-homepage/blob/db92a068a87e3d5b64226d6a0289924b0d24501c/.github/workflows/master.yml#L30-L51

    
      
          - name: Building nixos.org
            run: |
              nix build
              mkdir build
              cp -RL ./result/* ./result/.well-known/ ./build/
          
          
- name: Deploy to Netlify
            uses: nwtgck/actions-netlify@v1.2.3
            env:
              NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
              NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
            with:
              production-branch: 'master'
              production-deploy: ${{ github.ref_name == 'master' }}
              publish-dir: './build'
              github-token: ${{ secrets.GITHUB_TOKEN }}
              deploy-message: 'Deploy from GitHub Actions'
              enable-pull-request-comment: true
              enable-commit-comment: true
              enable-commit-status: true

This file has been truncated. show original

nomeata · February 16, 2022, 9:45am

Others have had success using portable nix in the netlify builder:

nomeata · February 16, 2022, 9:05pm

Small status update / experience report:

I was exploring the portable-nix route, and it indeed works. But my build times were 10min, four of which seem to be downloading nixpkgs and evaluating the derivation, then maybe 8 downloading all the build dependencies (granted, GHC and saxonb are not small), and then just a bit doing the actual work.

I tried to speed it up by using a remote builder (nixbuild.net), which was a bit of a hassle (ssh keys too large to fit in the secret env vars, so encrypting them in the repo and storing the decryption key in the env) but in the end that didn’t quite help: It seems that

./nix-portable nix-build --store ssh-ng://eu.nixbuild.net --option builders-use-substitutes true --max-jobs 0 --builders "ssh://eu.nixbuild.net x86_64-linux - 100 1 big-parallel,benchmark"

will still download all the build dependencies before kicking off the remote build, defeating part of the purpose of using a remote builder here. (This seems silly, the build dependencies are never needed locally – am I doing this wrongly)?

I might try to throw the nix cache into netlify’s cache, maybe that helps.

I just found out that another feature I was hoping to use in netlify doesn’t actually work, so maybe I’ll cease my netlify experiments, we’ll see.

nomeata · February 16, 2022, 10:34pm

Hmm, couldn’t get the netlify cache to work, failed for unclear reasons.

Is there really no simple way of building something on a remote builder without having to download build dependencies first? If there isn’t, is there already an issue at https://github.com/NixOS/nix/issues for this (I couldn’t find it).

DavHau · February 17, 2022, 7:11am

The nix remote build protocol seems to be a bit limited in several ways.
Since we have flakes now, you could express whatever you want to build as an output of a flake, then ssh to a remote machine, build that flake output there, then copy back the result + closure to your machine. That way you won’t download any build time deps.

nomeata · March 16, 2022, 10:36am

A recent change beta in on the nixbuild.net side allowed me to use the following script as the netlify build command, and get reasonable speed in building stuff:

#!/usr/bin/env bash
set -e

export NP_GIT=$(which git)

wget -nv https://github.com/DavHau/nix-portable/releases/download/v009/nix-portable
chmod +x nix-portable

mkdir -p ~/.ssh
openssl enc -d -aes-256-cbc -pbkdf2 -in netlify-nixbuild.ed25519.aes -out ~/.ssh/id_ed25519 -k "$NIXBUILD_SSH_KEY"
chmod 600 ~/.ssh/id_ed25519

echo 'eu.nixbuild.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPIQCZc54poJ8vqawd8TraNryQeJnvH1eLpIDgbiqymM' >> ~/.ssh/known_hosts

set -x

./nix-portable nix build -L --eval-store auto --store ssh-ng://eu.nixbuild.net -f default.nix


out="$(./nix-portable nix-store --query --outputs "$(./nix-portable nix-instantiate)")"
./nix-portable nix copy --no-check-sigs --from ssh-ng://eu.nixbuild.net "$out"

# The "result" symlink only valid inside the nix-portable sandbox
./nix-portable nix-shell -p bash --run "cp -rL $out output"

chmod -R u+w output

I’m not using it productively yet (mostly because I’m not sure I really like netlify and whether it offers everything I believe I need); maybe I’ll write a more detailed post once I do.

rickynils · March 16, 2022, 6:24pm

The recent change in nixbuild.net is the support for using --store to run builds without involving the local Nix store, and it was announced today: Lightning-fast CI with nixbuild.net

roberth · March 21, 2022, 10:34am

Hercules CI effects run sequentially, one job after another, where a job is like a commit, so there won’t be such a race condition.

nomeata · March 28, 2023, 6:18pm

I’ve been using this, including the nixbuild.net integration, for my personal website and works nice so far. Recently, I flakified the website generation, so now my build.sh script looks like this:

#!/usr/bin/env bash
set -e

export NP_GIT=$(which git)

wget -nv https://github.com/DavHau/nix-portable/releases/download/v009/nix-portable
chmod +x nix-portable

mkdir -p ~/.ssh
openssl enc -d -aes-256-cbc -pbkdf2 -in netlify-nixbuild.ed25519.aes -out ~/.ssh/id_ed25519 -k "$NIXBUILD_SSH_KEY"
chmod 600 ~/.ssh/id_ed25519

echo 'eu.nixbuild.net ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPIQCZc54poJ8vqawd8TraNryQeJnvH1eLpIDgbiqymM' >> ~/.ssh/known_hosts

set -x

./nix-portable nix build -L --eval-store auto --store ssh-ng://eu.nixbuild.net .#homepage


out="$(./nix-portable nix-store --query --outputs "$(./nix-portable nix path-info --derivation .#homepage)")"
./nix-portable nix copy --no-check-sigs --from ssh-ng://eu.nixbuild.net "$out"

# The "result" symlink only valid inside the nix-portable sandbox
./nix-portable nix run nixpkgs#coreutils -- --coreutils-prog=cp -rL "$out" output

chmod -R u+w output

mv -v output/netlify-functions/*.json netlify-functions/

Installing nix-portable installs an oldish, pre-2.7, version of nix where nix build . is understood differently, hence my use of an explicit output attribute in the script above.