Rate My Nextcloud Setup

Guides
1

Infra

I’m renting a VPS and a Storage Box from Hetzner.

Architecture

The VPS runs NixOS and mounts the Storage Box with SSHFS, and there is encfs on top of that for transparently encrypting all data stored on the box. It hosts the nextcloud data directory and automated daily backups.

Note: I don’t have a lot of experience with FUSE mounts but it seems like all the files will be owned by the user and group ID specified, regardless of who creates the files or any chown attempts. For this reason you must set the same user and group ID as what is used by FUSE mounts (maybe systemd?) for all users that run services which manipulate files on the mount. This solved all permission issues.

Container

Nextcloud runs in a NixOS container with Postgresql, Redis, and nginx. There is an nginx reverse proxy on the host OS. The NixOS container is ephemeral and uses bind mounts for persistent data. Although the host is not ephemeral, all the non-reproducible data is stored on /persistent or /mnt/box-plain with the exception of the SSH key used to mount the box. There is my user data in /home but there is nothing important there.

The container service is configured to run after the encfs mount service is done which runs after the sshfs mounting is done. This should ensure that after a reboot all the data is present for the container when it starts. The timeout for the container startup is also overwritten to 30 minutes.

Office

For online office support, Collabora CODE runs in an OCI container. The Nextcloud container is behind NAT, so I can have more services running parallelly in containers without unwanted interaction with the database, redis, or other services. The persistent data is separated for the container, so I can theoretically easily move a service to another VPS.

Maintenance

By default NixOS containers get a 1 minute timeout to start up with all their services which is not enough when there is an update to Nextcloud. I was unable to update nextcloud at all because of this and it took me forever to figure it out. With the custom timeout configuration updates are completely painless and there is basically no manual maintenance other than the occasional flake update and switching to the new nextcloud package when there is a new major release. I completely migrated all my data from OneDrive half a year ago and since then this setup has been a reliable and full-featured alternative.

Securing the Data

The storage box can only be accessed by SSH with an ssh key (no password) and cannot be accessed outside of the Hetzner network at all. To prevent the data at rest getting stolen everything on the external storage box is encrypted with a secure encfs configuration and in transit also by the sshfs layer under that. On top of that Nextcloud server side encryption is enabled, but after setting up encfs that seems a bit unnecessary. It makes it harder for attackers who gain access to the server to read the files but with a bit of skill and time they can be manually decrypted, given that the key can also be found on the server. There is also the zero trust Nextcloud E2EE feature which should be used for highly sensitive information.

To prevent the data getting lost there is a daily backup of the database and config - which are stored on the VPS block storage - to the external storage box. It’s a systemd service running every morning which creates a compressed archive with a timestamp. The storage box uses ZFS and has automated zero-cost backups as a built-in feature. This runs every morning after the other backup. Right now backing up the config, the database, and the data directory should cover everything that is necessary to recreate the setup if anything happens, except if the storage box loses the data, but for that I rely on the provider’s highly reliable setup.

Full disk encryption for the VPS block storage is still missing, which means that attackers with access to the physical disks or the hypervisors could steal and decrypt all the user data that is not secured with E2EE. I consider this a low risk scenario. The same can happen if they steal my log-in password, which - I hope - is unlikely.

Practical Info

It is a flake based config, here are all the modules for my VPS starting with conf.nix. The latest up to date state is not on main right now.

There are some things I would like to improve on:

  • encrypt the root partition and enable remote unlock over SSH in initrd
  • make the host ephemeral and put all the persistent files in /persistent including SSH keys
  • add more Nextcloud apps

This was my first time deploying any service on a VPS and I’m pretty happy with how it turned out. Suggestions, questions and critique are welcome!

3 Likes
2

Isn’t it already encrypted with SSHFS?

3

It’s encrypted in transit but not at rest as I understand.

5

Ah right now I got it.
If you want to protect your data from the provider or an attacker, the Nextcloud end to end encryption is probably your best bet since you don’t fully control the server.
This way even the server doesn’t know the data.

6

There’s actual E2E now? I only know of the builtin encryption module, that isn’t e2e though. To quote https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/encryption_configuration.html:

Encryption and decryption are performed on the Nextcloud server

In fact I even decrypted Nextcloud files by hand with the key files from a server once (because Nextcloud failed to decrypt everything on said server for… reasons. That’s a longer story).

This only makes sense if you use e.g. Amazon S3 as storage provider: then, you only store encrypted stuff in S3 and the keys are on the server under your control.

7

For a while now it seems.
But only on a per folder basis.

8

Unfortunately, this does not work for me. I get the same authorization problems as before. A very simple question. How does a network drive has to be mounted for “services.mysql.dataDir” to work, for example?

9

I don’t know what goes on under the hood in terms of sandboxing and permissions… I didn’t put the database on the network drive because it’s relatively small, just the next cloud data directory.

10

I know, but still I would prefer that anyone with physical access to the storage box can’t see any information. Right now there is still a directory structure, but the contents are encrypted. End to end is nice but it’s practical to have everything in there.

11

This is really frustrating and for me a showstopper. I want my important data on my redunant raid. My server has only a small ssd. The mysql database is the core of nextcloud. I also tried just to put the nextcloud.data on the network share. No success. But thanks anyway.

12

Databases really don’t like network shares because of locking issues that can lead to corruption.
When I looked into it a lot of guides recommended ISCSI or some Kubernetes grade storage.
Something like NFS or SMB is not recommended and I personally haven’t risked it yet myself.

1 Like
13

That is frustrating! Sorry

14

Another mildly frustrating thing, some apps don’t work, I get 404 errors in the console. Examples are bookmarks, polls, collectives… They show up in the toolbar but their icon doesn’t load and if I click on it I get an empty page with the nextcloud background.

15

nice! how are you deploying this now?

1 Like
16

Pet manual right now, just using SSH in the terminal

17

fair i see! i’ve been interested in deploying nixos to hetzner as well. i’m now tinkering a bit to do a declarative deploy there using terraform/opentofu.

1 Like
18

Nice, let me know if it works out! I’ve been looking at NixOps but maybe at a later point. What’s annoying is the initial setup because Hetzner doesn’t offer a NixOS image so I have to do a manual install via the command line. Perhaps there’s also a better way to do it with disko.

1 Like
19

i found hetzner configs both with and without disko, so maybe that can mostly add e.g. impermanence (if possible given in hetzner’s qemu setting?).

as mentioned on the wiki, it looks like some people have been using nixos-infect to install nixos there. that also seems the approach used by terranix-hcloud and teraflops to deploy nixos there now. for what it’s worth, here’s a few references on nixos on hetzner.

on hetzner images, they support hashicorp packer, which someone seems to have used with nixos there already before. a next step would seem for those deployment tools to switch to that.

2 Likes
20

I updated the post after improving my setup and fixing the issues I had with updating Nextcloud. Now all data on the external mount gets encrypted and the timing of the mounts and the container startup and permission issues are all fixed. The container startup timeout also needed to be increased. I also updated the link to the repo which I moved since the creation of this post.

@kiara it seems I forgot to reply to you. Did you ever get around to setting Nextcloud up?

@specter I know it’s been a long time, but if you want to give it another try with my setup, I think a lot of the issues are solved now by figuring out timing and permission problems. The important part is the container starting after all the external filesystems are mounted and the user IDs set up to match the fuse mounts. I still have my postgresql db on the block storage so I don’t know if it would work with the db on the mount but I don’t see why not.

21

@balint hey there, thanks for asking. i’ve been a bit busy and not very focused on nextcloud, so i don’t have one up yet. my focus had been closer to end-to-end reproducible environments, for which i’ve been playing with hetzner deployed by nixos-anywhere’s terraform module now.

for nextcloud, i’d like to look further into selfhostblocks, which seems like a nice way to further configure it.

thanks for updating your code as well. if i might ask about that, i wasn’t really familiar with encfs, so i was wondering, how did you decide in favor of that one over say traditional options like luks?