Hi,
I have the following configs:
90 ▎ # The service will run as this user
91 ▎ users.groups.systemd-journal-remote = { };
92 ▎ users.users.systemd-journal-remote = {
93 ▎ ▎ isSystemUser = true;
94 ▎ ▎ group = "systemd-journal-remote";
95 ▎ };
96 ▎ # Create the cert files here to be able to correctly set their permissions
97 ▎ environment.etc."ssl/certs/journal-certificate.pem" = {
98 ▎ ▎ source = ./certificate.crt;
99 ▎ ▎ mode = "0440";
100 ▎ ▎ user = "systemd-journal-remote";
101 ▎ };
102 ▎ environment.etc."ssl/ca/journal-trusted-ca.pem" = {
103 ▎ ▎ source = ./ca.crt;
104 ▎ ▎ mode = "0440";
105 ▎ ▎ user = "systemd-journal-remote";
106 ▎ };
107 ▎ age.secrets.certificateJournald = {
108 ▎ ▎ owner = "systemd-journal-remote";
┃ 109 ▎ ▎ mode = "440";
110 ▎ ▎ path = "/etc/ssl/private/journal-certificate.pem";
111 ▎ ▎ file = ./secrets/certificate.key.age;
112 ▎ };
113 ▎
114 ▎ services.journald = {
115 ▎ ▎ remote = {
116 ▎ ▎ ▎ enable = true;
117 ▎ ▎ ▎ listen = "https";
118 ▎ ▎ ▎ port = 19532;
119 ▎ ▎ ▎ settings = {
120 ▎ ▎ ▎ ▎ Remote = {
121 ▎ ▎ ▎ ▎ ▎ ServerKeyFile = "/etc/ssl/private/journal-certificate.pem";
122 ▎ ▎ ▎ ▎ ▎ ServerCertificateFile = "/etc/ssl/certs/journal-certificate.pem";
123 ▎ ▎ ▎ ▎ ▎ TrustedCertificateFile = "/etc/ssl/ca/journal-trusted-ca.pem";
124 ▎ ▎ ▎ ▎ };
125 ▎ ▎ ▎ };
126 ▎ ▎ };
127 ▎ };
128 ▎ systemd.sockets.systemd-journal-remote.socketConfig = {
129 ▎ ▎ ReceiveBuffer = "100M";
130 ▎ ▎ SendBuffer = "100M";
131 ▎ };
18 ▎ # The service will run as this user
19 ▎ users.groups.systemd-journal-upload = { };
20 ▎ users.users.systemd-journal-upload = {
21 ▎ ▎ isSystemUser = true;
22 ▎ ▎ group = "systemd-journal-upload";
23 ▎ };
24 ▎ # Create the cert files here to be able to correctly set their permissions
25 ▎ environment.etc = {
26 ▎ ▎ journal-certificate = {
27 ▎ ▎ ▎ source = ./certificate.crt;
28 ▎ ▎ ▎ mode = "0440";
29 ▎ ▎ ▎ user = "systemd-journal-upload";
30 ▎ ▎ };
31 ▎ ▎ trusted-ca = {
32 ▎ ▎ ▎ source = ../ca.crt;
33 ▎ ▎ ▎ mode = "0440";
34 ▎ ▎ ▎ user = "systemd-journal-upload";
35 ▎ ▎ };
36 ▎ };
37 ▎ age.secrets.certificateJournald = {
38 ▎ ▎ mode = "400";
39 ▎ ▎ owner = "systemd-journal-upload";
40 ▎ ▎ file = ./secrets/certificate.key.age;
41 ▎ };
42 ▎ services.journald = {
43 ▎ ▎ upload = {
44 ▎ ▎ ▎ enable = true;
45 ▎ ▎ ▎ settings = {
46 ▎ ▎ ▎ ▎ Upload = {
47 ▎ ▎ ▎ ▎ ▎ URL = "https://server:443";
48 ▎ ▎ ▎ ▎ ▎ ServerKeyFile = config.age.secrets.certificateJournald.path;
49 ▎ ▎ ▎ ▎ ▎ ServerCertificateFile = "/etc/journal-certificate";
50 ▎ ▎ ▎ ▎ ▎ TrustedCertificateFile = "/etc/trusted-ca";
51 ▎ ▎ ▎ ▎ };
52 ▎ ▎ ▎ ▎
53 ▎ ▎ ▎ };
54 ▎ ▎ };
55 ▎ };
This basic setup does seem to work to certain degree.
Unfortunately it did not take long for the upload service to start crashing consistently.
The error is:
Jul 24 18:34:48 client systemd-journal-upload[5034]: Buffer space is too small to write entry.
Jul 24 18:34:48 client systemd-journal-upload[5034]: Upload to https://server:443/upload failed: operation aborted by callback
A few lines get written on the server, but due to the high volume of logs, they will never catch up and be up to date.
Has anyone faced this before?
I have Loki and Promtail running as well, and they work without problems.