I run a nixOS server in Oracle’s A1 Ampare Always free tier that i use to test and run some services, I’m relaxed on security, I run a bunch of services that I expose to the internet like Minecraft, Luanti, openwebUI, openvscodeserver (that I access through a reverse-proxy with my ssh keys) and jupyter-lab.
But after updating from 24.11 to 25.05 I saw these appeared in my nixOS server:
Someone managed to breach and run their mining software.
I rebooted and they went away, but I’m afraid the server is compromised and I’ll have to reeinstalland change my ssh keys.
I supposed I wanted to ask if there is any way for me to know how they managed access? I couldn’t easily find anything in the logs, I don’t have password, just ssh login. It might have been through jupyter lab, I was running that under my own user, witch was the user they got access to, and was sharing the link with the token embedded with my collegues for a university project.
Thinking of reeinstalling with impermanence, I’m also thinking if there is any way the server could be very loud about attempted logins or suspicious logins?
If you don’t have any audit logging setup then you are blind. You should toast the server and assume all private key material on it is compromised. If you are going to expose random services to the internet I really recommend parking it all behind either ssh tunnels or a vpn. I’d also check all of your services with systemd-analyze security and if there are hardening options to apply do it, and ideally ship them upstream.
Thanks for the suggestions! I’ll try to take a look at potentially self hosting VPN solutions. I’ve skimmed through Security - NixOS Wiki, lynis seems like a good starting point for checking the security of the server with a lot of suggestions, even suggested the same one you said: systemd-analyze security.
As for audit logging setup, idk if you have any advice, especially for nixOS, on that? For truly secure logs, I’m thinking, I’d also probably have to send them off the server somewhere otherwise the attacker could potentially delete them
Thanks for your help!
Forgot to include in the post my config
It was using my personal user yeshey. Minecraft and Luanti have their own users, openwebUI has DynamicUser=True, I think the only service I had under my own user that was exposed was JupyterLab for our uni project, I didn’t have headspace to turn into it’s own declarative config, so I was just running from a tmux terminal, and just one person could have the link snooped from compromised cookies or what not and the server’s terminal is directly accessible from jupyterlab.
Still having some malware do that automatically, cuz I don’t think anyone would go through the manual trouble of accessing the server to put mining software running on such a weak machine so blatantly. The bot net is scary
Could also just have been one of your colleagues directly, FWIW.
If you have access to jupyter this isn’t hard, and it sounds like a manual attempt given they gave up, presumably when they realized making money off a random host isn’t that easy.
If you allow connecting via ssh with a password it could have been brute forced. I always set services.openssh.settings.PasswordAuthentication = false; to make this kind of attack useless.
