Pixelfed on NixOS Server

0x17 · July 29, 2024, 8:07am

I’m currently trying to get a pixelfed instance running on a NixOS server. I have the following simple code snippet in my config:

service.pixelfed = {
      enable = true;
      domain = "https://pixel.mydomain.space";
      secretFile = "/path/to/my/file";
};

Now when I run nixos-rebuild switch I get the error message:

pixelfed-data-setup.service: Failed to set up credentials: Protocol error

I have the right DNS entry for the Sub-Domain.
Did anyone have the same error or a solution for it or a tip where I can look further to find the error.

thx.

polygon · July 29, 2024, 8:12am

Try removing the https:// from the domain entry.

0x17 · July 29, 2024, 8:15am

still the same error without the https://

0x17 · July 29, 2024, 11:58am

Ok. I came a bit further.
I redo my whole .env file and play around with pixelfed-manage migrate:fresh
When I run this command I got the error:
Connection refused [tcp://127.0.0.1:6379]
while the attempt:
add_software_column_to_instance_table
This is also the error from my nixos-rebuild switch:

Jul 29 11:52:57 NB-ES-02 systemd[1]: Starting Pixelfed setup: migrations, environment file update, cache reload, data changes...
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681462]: sending incremental file list
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681462]: sent 1,151 bytes  received 33 bytes  2,368.00 bytes/sec
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681462]: total size is 14,515,784  speedup is 12,259.95
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:    INFO  Running migrations.
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   2021_08_30_050137_add_software_column_to_instances_table .......... 6ms FAIL
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]: In Connection.php line 829:
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:                                                                                
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'softwar
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   e' (Connection: mysql, SQL: alter table `instances` add `software` varchar(
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   191) null, add `user_count` int unsigned null, add `status_count` int unsig
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   ned null, add `last_crawled_at` timestamp null)
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:                                                                                
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]: In Connection.php line 587:
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:                                                                                
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'softwar
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:   e'
Jul 29 11:52:57 NB-ES-02 pixelfed-data-setup-start[681496]:                                                                                
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:    INFO  Running migrations.
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   2021_08_30_050137_add_software_column_to_instances_table .......... 7ms FAIL
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]: In Connection.php line 829:
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:                                                                                
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'softwar
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   e' (Connection: mysql, SQL: alter table `instances` add `software` varchar(
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   191) null, add `user_count` int unsigned null, add `status_count` int unsig
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   ned null, add `last_crawled_at` timestamp null)
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:                                                                                
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]: In Connection.php line 587:
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:                                                                                
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'softwar
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:   e'
Jul 29 11:52:58 NB-ES-02 pixelfed-data-setup-start[681541]:                                                                                
Jul 29 11:52:58 NB-ES-02 systemd[1]: pixelfed-data-setup.service: Main process exited, code=exited, status=1/FAILURE
Jul 29 11:52:58 NB-ES-02 systemd[1]: pixelfed-data-setup.service: Failed with result 'exit-code'.
Jul 29 11:52:58 NB-ES-02 systemd[1]: Failed to start Pixelfed setup: migrations, environment file update, cache reload, data changes.

I already wipe all the DBs. I am stuck

Edit: I start Redis manually. Now I got a “AUTH” failed error.

hexa · July 29, 2024, 2:27pm

There is a NixOS test, that I just tried and it worked on both 24.05 and unstable. The migrations run for close to 90s for me, and if your machine is a potato the systemd unit might run into an activation timeout.

github.com

NixOS/nixpkgs/blob/5627cbe0323cfa12c4d255ef3fba4a663715e85a/nixos/tests/web-apps/pixelfed/standard.nix

import ../../make-test-python.nix ({pkgs, ...}:
{
  name = "pixelfed-standard";
  meta.maintainers = with pkgs.lib.maintainers; [ raitobezarius ];

  nodes = {
    server = { pkgs, ... }: {
      services.pixelfed = {
        enable = true;
        domain = "pixelfed.local";
        # Configure NGINX.
        nginx = {};
        secretFile = (pkgs.writeText "secrets.env" ''
          # Snakeoil secret, can be any random 32-chars secret via CSPRNG.
          APP_KEY=adKK9EcY8Hcj3PLU7rzG9rJ6KKTOtYfA
        '');
        settings."FORCE_HTTPS_URLS" = false;
      };
    };
  };

This file has been truncated. show original

0x17 · July 29, 2024, 4:33pm

Ey! Don’t call my server a potato
I play around a bit and if I start Redis manually the error won’t appear

hexa · July 29, 2024, 7:22pm

So then the startup is racy, because the pixelfed-data-setup unit does not depend on the redis service, but requires it?

0x17 · July 29, 2024, 10:23pm

yes, I would say. It looks like pixelfed needs redis but doesn’t start it automatic by default

0x17 · July 30, 2024, 9:42am

TIL:
pixelfed got the option:

services.pixelfed.redis.createLocally = true;

now redis works as it should.
After some configuration with the pixelfed-manage tool (caching, migration etc.), all services were running and I can link nginx to the pixelfed-www directory.
However, nginx throws a 403 error.
I feel like I’m getting closer to success

***Edit (1): ***
At the moment I configure nginx via services.nginx.* Does it make a difference if I use nginx via the services.pixelfed.nginx.* options?

***Edit (2): ***
I tried the services.pixelfed.nginx.* option and it throws a 500 error. I will stay with the services.nginx.* option.

RaitoBezarius · August 2, 2024, 2:37pm

The services.pixelfed.nginx.* option sets up a bunch of things following upstream, I recommend sticking with it or sharing your NGINX fragment, otherwise, it’s difficult to debug.

0x17 · August 8, 2024, 10:10am

ok I made progress.
I started from the beginning with this config:

and now I use nginx within the pixelfed service as @RaitoBezarius suggested.

All services seems to be running:

The output from systemctl:

  UNIT                      LOAD   ACTIVE SUB     DESCRIPTION
  dbus.service              loaded active running D-Bus System Message Bus
  getty@tty1.service        loaded active running Getty on tty1
  mysql.service             loaded active running MySQL Server
  nginx.service             loaded active running Nginx Web Server
  nscd.service              loaded active running Name Service Cache Daemon (nsncd)
  phpfpm-pixelfed.service   loaded active running PHP FastCGI Process Manager service for pool pixelfed
  pixelfed-horizon.service  loaded active running Pixelfed task queueing via Laravel Horizon framework
  redis-pixelfed.service    loaded active running Redis Server - redis-pixelfed
  sshd.service              loaded active running SSH Daemon
  systemd-journald.service  loaded active running Journal Service
  systemd-logind.service    loaded active running User Login Management
  systemd-oomd.service      loaded active running Userspace Out-Of-Memory (OOM) Killer
  systemd-timesyncd.service loaded active running Network Time Synchronization
  systemd-udevd.service     loaded active running Rule-based Manager for Device Events and Files
  user@0.service            loaded active running User Manager for UID 0

Journactl for nginx:

Aug 08 09:55:30 MC-EX-01 systemd[1]: Reloading Nginx Web Server...
░░ Subject: A reload job for unit nginx.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A reload job for unit nginx.service has begun execution.
░░ 
░░ The job identifier is 12813.
Aug 08 09:55:31 MC-EX-01 nginx[23217]: nginx: the configuration file /nix/store/mf5y3cw5j8p2j5r7j8ckb2ggiv5d77a9-nginx.conf syntax is ok
Aug 08 09:55:31 MC-EX-01 nginx[23217]: nginx: configuration file /nix/store/mf5y3cw5j8p2j5r7j8ckb2ggiv5d77a9-nginx.conf test is successful
Aug 08 09:55:31 MC-EX-01 systemd[1]: Reloaded Nginx Web Server.
░░ Subject: A reload job for unit nginx.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A reload job for unit nginx.service has finished.
░░ 
░░ The job identifier is 12813 and the job result is done.

When I want to access pixelfed via browser I got a error 500.
Here in the systemctl from nginx:

XX.XX.XX.XXX - - [08/Aug/2024:09:58:47 +0000] "GET / HTTP/2.0" 500 1017 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"

Where do I have to look for this error?
When you open domain.tld/offline.html, the page displays normally and there is no automatic reload.

RaitoBezarius · August 8, 2024, 11:48am

Can you post journalctl -eu phpfpm-pixelfed.service ?

0x17 · August 8, 2024, 12:08pm

here it is:

Aug 08 08:49:46 MC-EX-01 php-fpm[6716]: [NOTICE] Terminating ...
Aug 08 08:49:46 MC-EX-01 systemd[1]: Stopping PHP FastCGI Process Manager service for pool pixelfed...
Aug 08 08:49:46 MC-EX-01 php-fpm[6716]: [NOTICE] exiting, bye-bye!
Aug 08 08:49:46 MC-EX-01 systemd[1]: phpfpm-pixelfed.service: Deactivated successfully.
Aug 08 08:49:46 MC-EX-01 systemd[1]: Stopped PHP FastCGI Process Manager service for pool pixelfed.
Aug 08 08:49:46 MC-EX-01 systemd[1]: phpfpm-pixelfed.service: Consumed 19.337s CPU time, 63.7M memory peak, 0B memory swap peak, no IO, no IP traffic.
Aug 08 08:52:01 MC-EX-01 systemd[1]: Starting PHP FastCGI Process Manager service for pool pixelfed...
Aug 08 08:52:02 MC-EX-01 php-fpm[22466]: [NOTICE] fpm is running, pid 22466
Aug 08 08:52:02 MC-EX-01 php-fpm[22466]: [NOTICE] ready to handle connections
Aug 08 08:52:02 MC-EX-01 systemd[1]: Started PHP FastCGI Process Manager service for pool pixelfed.
Aug 08 08:52:02 MC-EX-01 php-fpm[22466]: [NOTICE] systemd monitor interval set to 10000ms

RaitoBezarius · August 8, 2024, 12:11pm

The timestamps are inconsistent, the 500 happens at 09:58, the logs you just posted are from 08:52, the 500 should be logged on the application server in theory, or at least, NGINX should report the fpm error itself.

There’s not enough data to help right now.

0x17 · August 8, 2024, 12:52pm

these are the last entries in journalctl -.-
The times when error 500 happens do not appear in the nginx logs. where else can I look for it or how can I provocate log entries?

0x17 · August 9, 2024, 6:50am

Ok. I don’t do anything last night with the server and today I have the follow logs:

Aug 08 13:18:37 MC-EX-01 systemd[1]: Reloaded Nginx Web Server.
Aug 08 14:26:13 MC-EX-01 nginx[3640]: 2024/08/08 14:26:13 [error] 3640#3640: *197 access forbidden by rule, client: 45.148.10.59, server: p.kyb3r.space, request: "GET /.git/config HTTP/1.1", host: "monitor.nerdbude.com"
Aug 08 15:35:53 MC-EX-01 nginx[3640]: 2024/08/08 15:35:53 [error] 3640#3640: *245 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 64.227.189.179, server: p.kyb3r.space, request: "GET /alive.php HTTP/1.1", upstream: "fastcgi://unix:/run/phpfpm/pixelfed.sock:", host: "128.140.40.224"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *361 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.vscode/sftp.json HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *367 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.git/HEAD HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *368 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.svn/wc.db HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *364 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /wp-admin/setup-config.php HTTP/1.1", upstream: "fastcgi://unix:/run/phpfpm/pixelfed.sock:", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *383 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.ssh/id_ed25519 HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *386 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.env HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *391 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.kube/config HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *375 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.ssh/id_rsa HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *379 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /api/.env HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *382 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.env.production HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *387 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.ssh/id_ecdsa HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *394 access forbidden by rule, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /.aws/credentials HTTP/1.1", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *384 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /config/database.php HTTP/1.1", upstream: "fastcgi://unix:/run/phpfpm/pixelfed.sock:", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *371 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /config.php HTTP/1.1", upstream: "fastcgi://unix:/run/phpfpm/pixelfed.sock:", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *366 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /wp-config.php HTTP/1.1", upstream: "fastcgi://unix:/run/phpfpm/pixelfed.sock:", host: "p.kyb3r.space"
Aug 08 21:14:06 MC-EX-01 nginx[3640]: 2024/08/08 21:14:06 [error] 3640#3640: *363 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 213.232.87.230, server: p.kyb3r.space, request: "GET /phpinfo.php HTTP/1.1", upstream: "fastcgi://unix:/run/phpfpm/pixelfed.sock:", host: "p.kyb3r.space"
Aug 08 22:37:13 MC-EX-01 nginx[3640]: 2024/08/08 22:37:13 [error] 3640#3640: *441 access forbidden by rule, client: 66.94.109.92, server: p.kyb3r.space, request: "GET /.git/config HTTP/1.1", host: "128.140.40.224"
Aug 08 23:37:08 MC-EX-01 nginx[3640]: 2024/08/08 23:37:08 [error] 3640#3640: *472 access forbidden by rule, client: 66.94.109.92, server: p.kyb3r.space, request: "GET /.git/config HTTP/1.1", host: "128.140.40.224"
Aug 09 01:15:03 MC-EX-01 nginx[3640]: 2024/08/09 01:15:03 [error] 3640#3640: *518 access forbidden by rule, client: 66.29.134.3, server: p.kyb3r.space, request: "GET /.git/HEAD HTTP/1.1", host: "static.224.40.140.128.clients.your-server.de"
Aug 09 02:16:55 MC-EX-01 nginx[3640]: 2024/08/09 02:16:55 [error] 3640#3640: *646 access forbidden by rule, client: 94.156.68.162, server: p.kyb3r.space, request: "GET /.env HTTP/1.1", host: "128.140.40.224"
Aug 09 02:17:21 MC-EX-01 nginx[3640]: 2024/08/09 02:17:21 [error] 3640#3640: *647 access forbidden by rule, client: 94.156.68.162, server: p.kyb3r.space, request: "GET /.env HTTP/1.1", host: "128.140.40.224"
Aug 09 04:09:52 MC-EX-01 nginx[3640]: 2024/08/09 04:09:52 [error] 3640#3640: *687 access forbidden by rule, client: 45.89.247.57, server: p.kyb3r.space, request: "GET /.env HTTP/1.1", host: "128.140.40.224"
Aug 09 04:10:08 MC-EX-01 nginx[3640]: 2024/08/09 04:10:08 [error] 3640#3640: *688 access forbidden by rule, client: 45.89.247.57, server: p.kyb3r.space, request: "GET /.env HTTP/1.1", host: "128.140.40.224"
Aug 09 06:43:56 MC-EX-01 nginx[3640]: 2024/08/09 06:43:56 [error] 3640#3640: *1493 access forbidden by rule, client: 64.62.197.136, server: p.kyb3r.space, request: "GET /.git/config HTTP/1.1", host: "128.140.40.224"