Self Host Blocks - Building blocks for self-hosting with best practices

Announcements
1

Hi :wave:,

This post introduces a new project I’ve been working on: GitHub - ibizaman/selfhostblocks: Building blocks for self-hosting with battery included.

I have an Archlinux server where I self-host a bunch of services which are deployed through Emacs org-mode. I’m gradually migrating those services to two new servers - a cloud instance and an instance in my home - which are based on NixOS and deployed through disnix. While migrating, I intend to release a new project called Self Host Blocks. This project is to disnix what nixpkgs is to NixOS, a repo containing re-usable pieces that you can assemble to deploy to multiple inter-connected machines.

Although this project is very much in its infancy, I do use it to deploy a Vaultwarden instance behind Haproxy and with Keycloak and oauth2proxy as SSO on a Linode box.

I’m sharing it now because it’s better than later. I already punted this for long enough.I don’t expect anyone to use this just yet, not before an official V1. Indeed, I’m struggling right now after upgrading to NixOS 2, I have some changes to make in disnixos and I’m trying to be able to release disnixos. I’ll try to post weekly updates about my progress. I also intend to contribute to all projects I’m using along the way. Currently, I’m working on making disnixos handle network.nix files with arguments.

For now, I just want to let the community I’m working on this. Hopefully, that will spark some interesting discussions.

32 Likes
2

I still firmly believe this project to be important. But I’m changing a bit the implementation. This is for a few reasons.

  1. I’ve been struggling too much with Nixops 2. I’ve also read reports of users saying the project is not maintained anymore.
  2. The way disnix works, at least how I understand it, there’s no way to re-use all the services and options written by the community. Not verbatim at least. You must go through writing an additional layer on top of those to be able to use disnix. This is now IMO a fruitless effort and not worth pursuing.
  3. There’s some mismatch between Nixops 2 and disnix that require much more in depth knowledge of the codebase. But with the two bullets above, I’m not willing anymore to invest in this.

So I’m getting back to the basics. I’m switching to using colmena to deploy, I’m removing disnix from the codebase and I’m embracing nixpkgs modules which are a fantastic way to write configuration, like you all know.

The project selfhostblocks will get lighter thanks to this. It goes back to my core intention which was to provide an opinionated turnkey solution to self host services.

This is still very WIP but that’s what I just decided to do. I’ll post a new update when I’m done with that.

2 Likes
3

I’ve been processing quite a bit on this project. Recently I:

My TODO list right now :slight_smile: :

  • finish up the in progress items from above.
  • add actual integration tests to validate Authelia protection and for a bunch of other tests. Having none makes me nervous.
  • run the tests on CI
  • add a few more apps/services like gnucash
  • add module to replace CLI deploys for simple flake updated. The module would setup a gitea server where one would host at least their code that imports selfhostblocks. The idea would be that CI can then run on a schedule, run nix flake update, evaluate the config and present errors if any, if none it would create a PR showing the diff of the config with nix-diff and nvd and which when merged would actually deploy the update to the servers. I plan to make this CI script available as a GitHub action also.
5 Likes
4

I’m super happy to announce I added the first working example of a home assistant server backed by LDAP for user management that’s defined in 15 relevant lines of code and that’s deployable to a VM in 5 commands.

For now, unless a better idea, I’ll consider selfhostblocks v1 when I have a similar working example for all services handled.

EDIT: updated link

4 Likes
5

This might be a bit off topic, but I’m very curious how this worked now, because it sounds unlike anything I’ve seen before. o.o

1 Like
6

It sounds pretty similar to many not-too-complex deployments I have seen! Basically the stuff is of the type «start it and it will keep running», and you have a document with the proper invocations to start the stuff that needs starting, and maybe a bit of glue code to actually apply a bunch of snippetrs where they should be applied.

In a sense… think Automated Linux From Scratch, but runtime instead of buildtime, and less orderly.

7

@embr Haha, I’m happy to share but I must preface by it became unmaintainable pretty quickly.

Essentially I was using org-mode and org-babel in Emacs to execute commands on a remote host and tangle files (fancy word for generating files from a template) on the remote host. I was still using systemd to run any service. But installing them and configuring them required me running some code from Emacs.

I’ll add a bit more details when I’m back at my computer this weekend.

1 Like
8

I opened a Matrix room if anyone is interested. I’ll still post updates from time to time here. Link to Matrix room

2 Likes
9

All options are now published at https://shb.skarabox.com/
I’ll be working more on the docs now that they are visible :slight_smile:

1 Like
10

I’m super happy to share this. I just merged a PR that makes the SSL block that provides SSL certificates a true “block” in how I envisioned Self Host Blocks to work. Such a block:

  • Has a contract formalizing what that block’s output should be
  • Allows one to swap implementation without needing to change how the block is being used (here we have self-signed certificates as well as Let’s Encrypt generated certificates)
  • Has a NixOS VM test verifying that contract for all implementations (well, only for self-signed here as I didn’t figure out a great way to have tests with Let’s Encrypt, although I am using the Let’s Encrypt implementation of this block on my server)

The manual should have all the info needed to use that block SSL Block

4 Likes
11

Some of you may know I gave a talk at NixCon NA 24 about this project and specifically the contracts part. You can check the talk on the 8h long live stream but unfortunately part of the talk is lost there. I managed to trip on the power cord and the live stream stopped recording for 6 or 7 minutes at the end of the talk which is where I explain the solution to the problem I introduced at the beginning of the talk :man_facepalming:

The talk starts here https://www.youtube.com/live/G2uN7cw-PaU?si=D69ljsQhMgXz9dBV&t=1h14m11 but there’s the cut at 1:29:50.

Anyway, a LWN article just got out that goes over the talk and summarizes it quite well :blush: Nix at SCALE [LWN.net]

I still intend to write a more detailed blog post in the near future.

2 Likes
12

Finally took the time to write down what IMO makes contracts super useful Contracts I also documented the first and only contract existing yet SSL Generator Contract
Hopefully this is clear to read. Any feedback is truly appreciated!

3 Likes
13

Finally added a second documented contract to Self Host Blocks. The Backup Contract!

A service that can be backed by a block implementing this contract is the Vaultwarden Service.
And a block implementing this contract is the Restic Block.

This contract is still really in alpha status but it’s exciting!

3 Likes
14

Awesome, maybe I can finally implement my backblaze backup using this now.

1 Like
15

I’m actually working on exactly this! Restic backing up to both a local directory (in my case, in another ZFS pool) and to Backblaze.
Still a few rough edges I’m ironing as I write this. Specifically, the extraSecret part is failing to compile when used. I apologize I posted the above comment a day or two too soon.

1 Like
16

Alright, I’ve ironed out the last remaining issues and the backup block is working. I’ve been backing up Vaultwarden using Restic to a local directory (on another HDD) and to Backblaze thanks to the following config.

You’ll notice I don’t need to tell which directory to save nor which user to use thanks to the backup contract. The contract is essentially the snippet config.shb.vaultwarden.backup //.

shb.restic.instances."vaultwarden" = config.shb.vaultwarden.backup // {
  enable = true;

  passphraseFile = config.sops.secrets."vaultwarden/backup/passphrase".path;

  repositories = [
    {
      path = "/srv/backup/restic";
      timerConfig = {
        OnCalendar = "00:00:00";
        RandomizedDelaySec = "3h";
      };
    }
    {
      path = "s3:s3.us-west-000.backblazeb2.com/<mybucket>";
      timerConfig = {
        OnBootSec = "15min";
        OnUnitActiveSec = "1h";
        RandomizedDelaySec = "7min";
      };
      secrets = {
        AWS_ACCESS_KEY_ID.source = config.sops.secrets."backup/b2/access_key_id".path;
        AWS_SECRET_ACCESS_KEY.source = config.sops.secrets."backup/b2/secret_access_key".path;
      };
    }
  ];

  retention = {
    keep_within = "1d";
    keep_hourly = 24;
    keep_daily = 7;
    keep_weekly = 4;
    keep_monthly = 6;
  };
};

sops.secrets."vaultwarden/backup/passphrase" = {
  sopsFile = ./secrets.yaml;
  mode = "0400";
  owner = "vaultwarden";
  group = "vaultwarden";
};
sops.secrets."backup/b2/access_key_id" = {
  sopsFile = ./secrets.yaml;
  mode = "0400";
  owner = "vaultwarden";
  group = "vaultwarden";
};
sops.secrets."backup/b2/secret_access_key" = {
  sopsFile = ./secrets.yaml;
  mode = "0400";
  owner = "vaultwarden";
  group = "vaultwarden";
};

Also I updated the docs in the link I posted in the previous post.

3 Likes
17

I added a new contract for secrets. I’m slowly migrating all services and blocks to use it. The documentation is already written though and it provides the best explanation I could come up with for contracts so far Secret Contract

18

Speaking of contracts, I’ll be presenting them at NixCon 2024 in Berlin next week Enabling incremental adoption of NixOS with module contracts :: NixCon 2024 :: pretalx :smiling_face:

6 Likes
19

^ and the talk is now online. It was a super event and I met great people :blush: https://youtu.be/CP0hR6w1csc?si=hmneu-4SOHLy9wIM

9 Likes
20

I worked on implementing some of the contracts in the talk.

But first, there’s kind of tutorial now to get Self Host Blocks started for Nextcloud with backup and other features. If you follow these pages in order:

In that second page there’s a section for backup. You’ll see it uses a contract for backup and secrets. It means Nextcloud configures a backup program with what folders to backup in a generic way. Then on Restic’s side, it understands these generic options and configures the nixpkgs Restic service accordingly.

There’s a similar database backup contract but for backing up databases. This time, PostgreSQL itself instructs a generic backup program how it should be backed up. The difference with the above backup contract is that here, it’s not just files on disk but it’s running a command to generate the backup. Then again Restic understands these generic options and configures the nixpkgs Restic service accordingly.

I could’ve merged both contracts into one but it’s better to keep the contracts as small as possible. It makes them more composable and more testable. It’s the same idea being Go interfaces.

Indeed, because the contracts are generic, there are generic tests! For example the backup contract generic one makes sure any module claiming to implement the backup contract works the same. It is then used to test the Restic implemention.
Same for the database backup contract generic test which is used to test the Restic implementation on a Postgres database.

So for something else to support backing up using the backup and database backup contract, it should be sufficient to pass those generic tests.

And I have the same for secrets now.
The contract, an implementation by SOPS and a hardcoded implementation to help testing. It’s now used in a lot of modules in SHB like authelia. Then the generic test that is used to test the hardcoded implementation.

7 Likes