Self Host Blocks - Building blocks for self-hosting with best practices

Hi :wave:,

This post introduces a new project I’ve been working on: GitHub - ibizaman/selfhostblocks: Building blocks for self-hosting with battery included.

I have an Archlinux server where I self-host a bunch of services which are deployed through Emacs org-mode. I’m gradually migrating those services to two new servers - a cloud instance and an instance in my home - which are based on NixOS and deployed through disnix. While migrating, I intend to release a new project called Self Host Blocks. This project is to disnix what nixpkgs is to NixOS, a repo containing re-usable pieces that you can assemble to deploy to multiple inter-connected machines.

Although this project is very much in its infancy, I do use it to deploy a Vaultwarden instance behind Haproxy and with Keycloak and oauth2proxy as SSO on a Linode box.

I’m sharing it now because it’s better than later. I already punted this for long enough.I don’t expect anyone to use this just yet, not before an official V1. Indeed, I’m struggling right now after upgrading to NixOS 2, I have some changes to make in disnixos and I’m trying to be able to release disnixos. I’ll try to post weekly updates about my progress. I also intend to contribute to all projects I’m using along the way. Currently, I’m working on making disnixos handle network.nix files with arguments.

For now, I just want to let the community I’m working on this. Hopefully, that will spark some interesting discussions.

22 Likes

I still firmly believe this project to be important. But I’m changing a bit the implementation. This is for a few reasons.

  1. I’ve been struggling too much with Nixops 2. I’ve also read reports of users saying the project is not maintained anymore.
  2. The way disnix works, at least how I understand it, there’s no way to re-use all the services and options written by the community. Not verbatim at least. You must go through writing an additional layer on top of those to be able to use disnix. This is now IMO a fruitless effort and not worth pursuing.
  3. There’s some mismatch between Nixops 2 and disnix that require much more in depth knowledge of the codebase. But with the two bullets above, I’m not willing anymore to invest in this.

So I’m getting back to the basics. I’m switching to using colmena to deploy, I’m removing disnix from the codebase and I’m embracing nixpkgs modules which are a fantastic way to write configuration, like you all know.

The project selfhostblocks will get lighter thanks to this. It goes back to my core intention which was to provide an opinionated turnkey solution to self host services.

This is still very WIP but that’s what I just decided to do. I’ll post a new update when I’m done with that.

2 Likes

I’ve been processing quite a bit on this project. Recently I:

My TODO list right now :slight_smile: :

  • finish up the in progress items from above.
  • add actual integration tests to validate Authelia protection and for a bunch of other tests. Having none makes me nervous.
  • run the tests on CI
  • add a few more apps/services like gnucash
  • add module to replace CLI deploys for simple flake updated. The module would setup a gitea server where one would host at least their code that imports selfhostblocks. The idea would be that CI can then run on a schedule, run nix flake update, evaluate the config and present errors if any, if none it would create a PR showing the diff of the config with nix-diff and nvd and which when merged would actually deploy the update to the servers. I plan to make this CI script available as a GitHub action also.
5 Likes

I’m super happy to announce I added the first working example of a home assistant server backed by LDAP for user management that’s defined in 15 relevant lines of code and that’s deployable to a VM in 5 commands.

For now, unless a better idea, I’ll consider selfhostblocks v1 when I have a similar working example for all services handled.

EDIT: updated link

3 Likes

This might be a bit off topic, but I’m very curious how this worked now, because it sounds unlike anything I’ve seen before. o.o

1 Like

It sounds pretty similar to many not-too-complex deployments I have seen! Basically the stuff is of the type «start it and it will keep running», and you have a document with the proper invocations to start the stuff that needs starting, and maybe a bit of glue code to actually apply a bunch of snippetrs where they should be applied.

In a sense… think Automated Linux From Scratch, but runtime instead of buildtime, and less orderly.

@embr Haha, I’m happy to share but I must preface by it became unmaintainable pretty quickly.

Essentially I was using org-mode and org-babel in Emacs to execute commands on a remote host and tangle files (fancy word for generating files from a template) on the remote host. I was still using systemd to run any service. But installing them and configuring them required me running some code from Emacs.

I’ll add a bit more details when I’m back at my computer this weekend.

1 Like

I opened a Matrix room if anyone is interested. I’ll still post updates from time to time here. Link to Matrix room

1 Like

All options are now published at https://shb.skarabox.com/
I’ll be working more on the docs now that they are visible :slight_smile:

1 Like

I’m super happy to share this. I just merged a PR that makes the SSL block that provides SSL certificates a true “block” in how I envisioned Self Host Blocks to work. Such a block:

  • Has a contract formalizing what that block’s output should be
  • Allows one to swap implementation without needing to change how the block is being used (here we have self-signed certificates as well as Let’s Encrypt generated certificates)
  • Has a NixOS VM test verifying that contract for all implementations (well, only for self-signed here as I didn’t figure out a great way to have tests with Let’s Encrypt, although I am using the Let’s Encrypt implementation of this block on my server)

The manual should have all the info needed to use that block SSL Block

4 Likes

Some of you may know I gave a talk at NixCon NA 24 about this project and specifically the contracts part. You can check the talk on the 8h long live stream but unfortunately part of the talk is lost there. I managed to trip on the power cord and the live stream stopped recording for 6 or 7 minutes at the end of the talk which is where I explain the solution to the problem I introduced at the beginning of the talk :man_facepalming:

The talk starts here https://www.youtube.com/live/G2uN7cw-PaU?si=D69ljsQhMgXz9dBV&t=1h14m11 but there’s the cut at 1:29:50.

Anyway, a LWN article just got out that goes over the talk and summarizes it quite well :blush: Nix at SCALE [LWN.net]

I still intend to write a more detailed blog post in the near future.

2 Likes

Finally took the time to write down what IMO makes contracts super useful Contracts I also documented the first and only contract existing yet SSL Generator Contract
Hopefully this is clear to read. Any feedback is truly appreciated!

3 Likes