Unable to start cron?

tolgaerok · September 15, 2023, 8:14pm

Hi all i have this in my nix file but its not working. i made a system-wide package to archive my nixos folder and called it nixos-archive but the cron job aint working

Ive tried manually running and it works: sh -c “nixos-archive >> /home/tolga/test.log run” and logs the output

#---------------------------------------------------------------------
  # Back up nixos folder every 1 min (testing)
  #---------------------------------------------------------------------
  services.cron = {
    enable = true;
    systemCronJobs = [

      "*/1 * * * * nixos-archive >> /home/tolga/test.log run"

    ];
  };

Its set to 1 minute purely for testing purposes.

[tolga@HP-G800:/etc/nixos]$ sudo systemctl status cron.service


[sudo] password for tolga: 
● cron.service - Cron Daemon
     Loaded: loaded (/etc/systemd/system/cron.service; enabled; preset: enabled)
     Active: active (running) since Sat 2023-09-16 03:49:26 AWST; 20min ago
   Main PID: 124323 (cron)
         IP: 0B in, 0B out
         IO: 20.0K read, 0B written
      Tasks: 1 (limit: 33533)
     Memory: 220.0K
        CPU: 27ms
     CGroup: /system.slice/cron.service
             └─124323 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron -n

Sep 16 03:52:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[124773]: (tolga) MAIL (mailed 122 bytes of output but got status 0x0001
                                                                                               )
Sep 16 03:53:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[124868]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)
Sep 16 03:54:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[124935]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)
Sep 16 03:55:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[124949]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)
Sep 16 03:56:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[125164]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)
Sep 16 03:56:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[125163]: (tolga) MAIL (mailed 122 bytes of output but got status 0x0001
                                                                                               )
Sep 16 03:57:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[125320]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)
Sep 16 03:58:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[125636]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)
Sep 16 03:59:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[124323]: (*system*) RELOAD (/etc/crontab)
Sep 16 03:59:01 HP-G800 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron[126824]: (tolga) CMD (sudo nixos-archive >> /home/tolga/test.log)

sudo cat /etc/crontab

SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash
PATH=/nix/store/ffz95zaxqb2kmcb7mbl969598vchl133-system-path/bin:/nix/store/ffz95zaxqb2kmcb7mbl969598vchl133-system-path/sbin

NIX_CONF_DIR=/etc/nix
*/1 * * * * sudo nixos-archive >> /home/tolga/test.log run

ps aux | grep cron

root      124323  0.0  0.0   5508  1920 ?        Ss   03:49   0:00 /nix/store/42sfnspzz0s48zlvym40f1w1j5ihdnfg-cron-4.1/bin/cron -n
tolga     127932  0.0  0.0   6512  2432 pts/0    S+   04:17   0:00 grep cron

colemickens · September 15, 2023, 8:36pm

I’m a bit confused. It looks like cron is (trying to call/)calling that command every 1 minute…

Thoughts:

Should one expect that to work in crontab, versus doing bash -c "sudo nixos-archive >> /wherever.log"? Also, is nixos-archive installed system-wide such that that command is available to cron?
Maybe try redirecting stderr to your log too, just in case?
I’d probably just go ahead and write a systemd timer/service for this.
It’s interesting that you don’t have sudo in the command invocation in your nixos/cron configuration, let it seems like cron is calling it with sudo? Is there configuration drift? Or maybe cron just does that?

tolgaerok · September 15, 2023, 8:48pm

I did state: Its set to 1 minute purely for testing purposes.

i noticed there was no crontab set up so i injected my cron task into it by running: crontab -e
and added */59 * * * * nixos-archive >> /home/tolga/test.log

before changing to 59 mins, i successfully got it to work every 1 minute:

I made a short script to inject the line of code into crontab incase a new install awaits!

TLATER · September 15, 2023, 9:26pm

Your configuration should be creating one in /etc/crontab: https://github.com/NixOS/nixpkgs/blob/360a7d31c30abefdc490d203f80e3221b7a24af2/nixos/modules/services/scheduling/cron.nix#L103

Long shot, since you’re saying it was resolved by manually editing the crontab (which I assume puts one in /var/cron, but I’ve not used cron in so long I’ve forgotten how it worked), but did you consider whether nixos-archive is present in cron’s environment by default?

Maybe you needed:

services.cron = {
    enable = true;
    systemCronJobs = [
      "*/1 * * * * ${nixos-archive-derivation}/bin/nixos-archive >> /home/tolga/test.log run"
    ];
  };

Not sure why that wouldn’t work if a manually created crontab does work, though.

I’m also wondering if that run at the end was a typo and/or if that caused your issue.

Always saddened when people write scripts they run on installation because they can’t figure out NixOS config, that’s kind of like defying the purpose of the distro. At the very least you could put something in activationScripts instead of manually running it, but oh well.

Systemd is definitely nicer than cron, by the way:

systemd = {
  services.nixos-archive = {
    path = [
      nixos-archive-derivation # Assuming you don't just put the script contents in `script`
    ];
    script = "nixos-archive";
    description = "NixOS backups";
    serviceConfig.Type = "oneshot";
  };

  timers.nixos-archive = {
    timerConfig.OnCalendar = "hourly";
  };
};

Then you can get your logs from journalctl --unit nixos-archive, instead of having to dump things into a log directory in your home.

tolgaerok · September 16, 2023, 4:26pm

I made a system wide script called nixos-archive

nixos-archive = pkgs.writeScriptBin “nixos-archive” ‘’ bla bla bla

tolgaerok · September 16, 2023, 4:35pm

Always saddened when people write scripts they run on installation because they can’t figure out NixOS config, that’s kind of like defying the purpose of the distro. At the very least you could put something in activationScripts instead of manually running it, but oh well.

Why say such nonsense. We all have different ways of learning at our own pace. On the contrary it saddens to read such comments in linux forums

TLATER · September 16, 2023, 8:59pm

I’m genuinely sorry this came across thay way, this was not meant at all to be an insult. I can just see you’re so close, and was a bit sad because it felt like I’d lost the opportunity to point the problem out to you. There’s nothing wrong with learning at all, on the contrary, I try my best to help with that here!

In the spirit of that, let me first explain what I think the problem is, and how you can work around these kinds of problems without needing a separate script if you do find yourself stuck for whatever reason:

So, “system-wide” on NixOS usually means that, for all users, the application will be added to $PATH. NixOS does not have a /usr/bin like other distros where you would find all “system-wide” binaries, at least not exactly the same way.

Normally, to make something system-wide, you would use that definition something like this:

let
  nixos-archive = pkgs.writeScriptBin "nixos-archive" "bla bla bla";
in
  environment.systemPackages = [
    nixos-archive
  ];

I think so far none of this should be surprising.

However, systemd units, and therefore also your cron process as it’s started with a systemd unit, do not have the contents from environment.systemPackages in their $PATH. This means that “system-wide” packages won’t be available from your crontab by default.

Hence you need something like this:

TLATER:

let
  nixos-archive = pkgs.writeScriptBin "nixos-archive" "bla bla bla";
in services.cron = {
    enable = true;
    systemCronJobs = [
      "*/1 * * * * ${nixos-archive}/bin/nixos-archive >> /home/tolga/test.log"
    ];
  };

Where ${nixos-archive} will be substituted with an absolute path to your derivation, so that your cron process doesn’t need to have the system-wide $PATH in its environment.

This is where I’m still a bit surprised that your manual crontab did work, since it should not have changed anything about whether cron can find your binary. Perhaps it does have access to environment.systemPackages somehow, and the only issue was a typo.

Or maybe the crontab editor captures your users’ environment somehow, in which case that entry may break next time there’s a bash update. More likely, it’s in a user crontab now, and crontab runs user-specific shell init before executing the entry so that surprises like $PATH not matching don’t happen.

With that explained, I definitely understand that, occasionally, something will be frustrating enough that you just want to put the files in the correct place by hand during installation.

NixOS has specific ways of doing that so you don’t need to write a separate script to execute:

Firstly, you can use environment.etc, which is the easier of the two, but can only write files to /etc:

environment.etc."crontab".text = ''
  */59 * * * * nixos-archive >> /home/tolga/test.log
'';

If that still doesn’t give you enough power, you can always use system.activationScripts to run any code you want as part of the nixos-rebuild process:

system.activationScripts.crontab = {
  deps = [ "etc" ];
  text = ''
    cp ${./hand-written-crontab} /etc/crontab
  '';

The deps thing here just instructs NixOS to run that script after etc was created, to make sure that the activation script for creating /etc ran first. It’s not always necessary, but prevents race conditions.

This way, you should never need a post-installation setup script, they can just be run as part of system installation!

tolgaerok · September 19, 2023, 5:26am

Apologies accepted, thank-you!

Would i be correct to say:

let
  nixos-archive = pkgs.writeScriptBin "nixos-archive" "
    #!/bin/bash
    # Personal nixos folder archiver backup
    # Tolga Erok. ¯\_(ツ)_/¯
    # 9/9/2023

    # Define the backup folder path within /etc/nixos
    backup_folder="/etc/nixos/NIXOS-ARCHIVES"

    # Get the current date and time in the required format
    current_date=$(date +"%Y %b %a, %l:%M%p")
    backup_subfolder=$(date +"%Y/%b/%a,%l:%M%p")

    # Create the backup folder structure if it doesn't exist
    mkdir -p "$backup_folder/$backup_subfolder"

    # Define the backup filename without extension
    backup_filename=$(date +"%a,%l:%M%p")

    # Zip the contents of /etc/nixos without the folder structure
    zip -r "$backup_folder/$backup_subfolder/$backup_filename.zip" /etc/nixos/* -x "/etc/nixos/NIXOS-ARCHIVES/*"

    echo "Backup completed and stored in $backup_folder/$backup_subfolder/$backup_filename.zip
  '';
in {
  #---------------------------------------------------------------------
  # Type: nixos-archive in terminal to execute above bash script
  #---------------------------------------------------------------------

  environment.systemPackages = [ nixos-archive ];

  services.cron = {
    enable = true;
    systemCronJobs = [
      "*/1 * * * * ${nixos-archive}/bin/nixos-archive >> /home/tolga/test.log"
    ];
  };
}

Then remove the entry in crontab -e */59 * * * * nixos-archive >> /home/tolga/test.log

At the moment, it’s doing what I want it to do for now, but i totally agree with you that perhaps the best practice is via systemd?

Not entirely certain what you mean here: cp ${./hand-written-crontab} /etc/crontab

tolgaerok · September 19, 2023, 6:08am

There again, i may change the script to do backups when there is a difference between the source directory and the backup directory?

TLATER · September 19, 2023, 7:40am

Yep! That’s exactly what I was trying to say

One comment on that script, does zip actually work? I don’t think it will, because that won’t be in $PATH either. You should check if your archives actually contain any data, they might just have error messages in them.

You may also want to add set -eu immediately after your starting comments so that the script properly fails if it doesn’t work instead of tricking you into thinking it succeeded. Bash is annoying like that.

Also, to be clear about how this works, nix will translate:

${nixos-archive}/bin/nixos-archive` => `/nix/store/nixos-archive-<hash>/bin/nixos-archive`,

And then place the script you defined with writeScriptBin in that path.

Using absolute paths like this is often necessary, because NixOS does not have standard fhs paths like /bin, so doesn’t know where to find nixos-archive unless you use an absolute path. Only specific things, like your terminal (and I guess user crontabs?), have some extra code to make that work.

I'd argue yes

Systemd has a built-in replacement for cron (timers), and is at least partially intended to be a replacement for it.

Systemd has more features, is arguably simpler to understand because its configuration is human readable (INI-style) and the journal logging makes it more user friendly than cron. It’s also easier to maintain since you don’t need to rotate your logs manually…

IMO there is little reason to use cron over systemd these days. The main reasons I’ve heard are:

You’re on an embedded system that can’t afford the extra disk space for systemd
- Moot point on NixOS, you can’t use NixOS without systemd
You don’t want to rewrite your existing crontabs
You’re an old dog and don’t want to learn new tricks
You personally dislike Lennart Poettering for some reason and don’t use any software he’s involved in in protest

Each of those arguments is valid in its own way, but I wouldn’t build best practices for general purpose, modern Linux desktop use on them.

That’s not to say it’s wrong to use cron, it just feels a bit superfluous, when systemd timers are already installed and available.

That is me trying to explain NixOS a little, while giving you a tip on post-installation scripts.

Under the hood, every single file installed on NixOS that is not inside /nix/store is created like that.

The activation scripts are run every time you do one of the following:

Run nixos-rebuild switch
Boot NixOS

Activation scripts do a lot of things, among them creating important directories like /etc and creating symlinks to all kinds of important files stores in the nix store.

You can basically think of them as the installation instructions for NixOS - NixOS reinstalls itself every time you boot, without deleting your data files.

So if you ever want manually to run a script after installation again, consider writing an activationScript like the one I suggest instead.

Feel free to ask me if you ever need help with that.

You could consider that. Commands like rsync or the more advanced casync might help with that.

That said, if you start adding complex features like this, maybe you should look into existing tools instead. You might like borg backup or restic.