So I almost have it working thanks to some LLM usage but I’m still running into weirdness when trying to use it over SSH using GitLab CI.
I have a key on the servers and the key in GitLab (in a secure as possible way) but I still get host key errors when trying to run it on the server itself over SSH. It works fine if I run it locally with the key loaded.
Could you clarify what specific type of host key error occurs? Is the target host key not known or is the target rejecting your key?
You will generally have to supply the public host key (use to fetch it ssh-keyscan) ahead of time via a CI variable to bypass TOFU for known hosts. This might explain the difference between local/CI environment where you already have trusted the host key in the past (interactively, TOFU).
For nixos-rebuild switch I do the following with keys/ssh/servers/boron.pub being the checked-in output of ssh-keyscan -p 2077 boron.nebula.arasaka.io:
So I have the public ssh key defined in the configuration on the system that I’m deploying to and have that key available to the GitLab CI. I get this error:
No ED25519 host key is known for thelio-04 and you have requested strict checking.
Host key verification failed.
error: failed to start SSH connection to 'deploy@thelio-04'
That means the ~/.ssh/known_hosts file doesn’t contain a key for the host you’re uploading to, so ssh refuses to upload under the assumption that you might be signing into a honeypot (and potentially uploading confidential data to somewhere you didn’t intend to, even if the connection is secure).
This is the relevant bit from @Cobalt 's snippets that solves that:
As @Cobalt says, locally you probably at some point blindly typed the y for accepting a host key, but gitlab CI - sensibly - is a bit stricter about this.
If you don’t share the verbatim config it’s hard to help more than explain the error
I will say that I personally also would recommend a pull-style config (potentially using colmena) for CI purposes (perhaps triggered by a webhook or long polling). That way CI can stay much simpler, you can rely on HTTPS instead of having to manually do the key exchange, and you can avoid adding an ssh key to a web service.
git+ssh works fine and uses the ssh agent of the user caling nix (so just adding a key to the .ssh/config and agent works). As an example, see this public repo using private ones as inputs. I don’t think this works well for fetchFromGitLab for derivation sources inside nix expressions though.
git over HTTP with the job token (and adjusted permissions for it) is probably more stable though and can be configured like a normal access token.
So I can run commands to the systems over ssh currently so that is fine really. Currently I run the job on the system I will update, copy the config.nix from the runner to the system and build locally. It works but I feel like it is not great.
Could you clarify what your actual requirement/usecase here is, please?
Do you want to run deploy-rs against a remote server from a CI job or do you run deploy-rs against localhost on the remote server (initiated from a CI job)?
My message was in regard to using private GitLab repositories with nix (which are usual for GitLab instances with global visibility restrictions).
If you run the actual nix interpreter/build on the target then the same guidance for access to them would still apply (if you are able to reach the GitLab server). The only minor difference would be that you would probably use a PAT/GAT instead of a job token then.
I might be info dumping here too much. I have done deploy-rs with CI and private GitLab instances before but some of that might be too specific too my requirements.
Each job is for a certain build server which will download the config.nix from the repo (since it is locally in the gitlab copy and copy it to /etc/nixos/
Then nixos-rebuild switch is ran
It works well enough but seems iffy sometimes like it doesn’t actually use the newest config to switch to (for example I’ll see an older user account readded back or it is using an old ssh key for a user).