NixOS on Free Oracle Cloud Arm A1

Hi!
I wanted to share my experience in setting up NixOS in a free Oracle VM.Standard.A1.Flex Instance (max. 4 ARM Cores with max 24 GB RAM & max. 200 GB - all forever-free).

First of all the result:

[root@vm:~]# uname -a
Linux vm 5.10.94 #1-NixOS SMP Thu Jan 27 09:54:36 UTC 2022 aarch64 GNU/Linux

Most of all it worked out pretty well, just some tiny things that took hours to figure out.

I followed this guide (even though aarch64 is never mentioned, it worked): Install NixOS on a Server With a Different Filesystem - NixOS Wiki

My Problems - most probably accidental:

  • The VM took very long (5-10 minutes?) to execute kexec and reboot - I didn’t expect that and thought it didn’t work
  • Somehow my ssh-key wasn’t added to the kexec’ed system (maybe a typo by me?) - but I could attach a virtual serial console to debug the problem and add it (also helps dramatically for debugging other problems)
  • Seperating /boot and /efi to different partitions didn’t work for me
    File system "/dev/block/8:1" has wrong type for an EFI System Partition (ESP). (but this should be /boot and not /boot/efi)
  • resizing Partition 15 (ESP) didn’t work out either - segfault in fatresize
    => recreating a new /boot partition that also hosts EFI

I’m very happy with my free VM :blush:
Thomas131

6 Likes

How much outbound traffic is allowed in the free forever plan?

GCP allows 1GB per month

1 Like

If I read correctly, 10 TB.

Cloud Free Tier | Oracle - 3rd column

2 Likes

@Thomas131 Thanks for reporting your success.

Checking out these Ampere VMs was also on my list. Setting one up with NixOS was pretty smooth:

I started with an Ubuntu Image and ssh’ing in to the ubuntu user I installed nix and built the kexecstuff as per Install NixOS on a Server With a Different Filesystem - NixOS Wiki directly on the VM. tar xf the resulting tarball into a clean subdir of my ~ and run it.
Waiting for a ping reply did not work (never got one), but when I tried ssh root@... it immediately connected on first attempt.
Seeing that I had two partitions available, one for the rootfs and one for EFI, I simply did

zpool create -O compress=on -O mountpoint=legacy tank /dev/sda1 -f
zfs create -o xattr=off -o atime=off tank/nix
mount -t zfs tank /mnt
mkdir /mnt/boot /mnt/nix
mount -t zfs tank/nix /mnt/nix
mount /dev/sda15 /mnt/boot

No BIOS boot partition required. I also skipped creating a swap partition, having 24GB of RAM available.

Then is set my nixpkgs channel to nixos-21.11-aarch64, updated it
and started the install. My initial configuration.nix looked like:

(Note: scp and a local editor are useful as you can also backup your configuration.nix this way.)

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.supportedFilesystems = [ "zfs" ];
  boot.kernelParams = [ "net.ifnames=0" ];

  boot.zfs.devNodes = "/dev";

  networking.hostId = throw "set your own";	
  networking.hostName = throw "set your own"; # Define your hostname.

  networking.useDHCP = false;  # deprecated flag, set to false until removed
  networking = {
    defaultGateway = "10.0.0.1";
    nameservers = [ "9.9.9.9" ];  
    interfaces.eth0 = {
      ipAddress = throw "set your own";
      prefixLength = 24;
    };
  };

 
  # Select internationalisation properties.
   console = {
     font = "Lat2-Terminus16";
     keyMap = "de";
   };
   i18n = {
     defaultLocale = "de_DE.UTF-8";
   };

  # Set your time zone.
   time.timeZone = "Europe/Berlin";
  
  users.users.root.openssh.authorizedKeys.keys = [
    "set your own key here"
  ];

  environment.systemPackages = with pkgs; [
     vim # Do not forget to add an editor to edit configuration.nix! The Nano editor is also installed by default.
     screen
  ];

  # Enable the OpenSSH daemon.
  services.openssh.enable = true;

  networking.firewall.enable = true;
  networking.firewall.allowPing = true;	

  nix.autoOptimiseStore = true;	

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "22.05"; # Did you read the comment?

}

Most time was spent going through the Cloud Console to fix a typo I had made in the default gateway IP, rendering the machine unreachable via ssh.

The process should easily be scriptable, taking only IP address of the VM (set up with Ubuntu) and a ssh key as parameters.

1 Like

Thanks for the reply and trying it out!

I just noticed that nixOS doesn’t use Grub and therefore doesn’t need a big boot-partiton :see_no_evil: So all my mess trying to resize the original EFI-Partiton was useless :rofl:

I just did a similar installation on a similar VM. I used the NIXOS_LUSTRATE approach described at NixOS 23.11 manual | Nix & NixOS, skipping steps 5-8 as I was replacing the original distro.

The only trick was that the Ubuntu image mounts the EFI partition (/dev/sda15 in my case) at /boot/efi, but I just mounted it at /boot and I was able to run nixos-generate-config --root /, bootctl install and then nixos-install --root /. After a reboot, I just had to rm -rf /old-root and it’s done!

2 Likes

It turns out the approach in my previous post only mostly works: it leaves you with a boot partition that’s way too small. I documented another approach here: Nixos on an Oracle Free Tier Ampere machine :: korfuri's website

4 Likes

I just tried this myself, it works great! Thanks for the clear tutorial.

A couple of quick notes I should add after running through it:

  • On the free tier, as you mentioned, you can get 4 cores, 24 GB RAM, and 200 GB disk. I will add that for disk volume performance you can select up to 20 VPU (high performance).
  • After git cloning cleverca22/nix-tests.git, you’ll need to cd nix-tests/kexec.
  • For the untar step, the file is nixos-system-aarch64-linux.tar.xz and not nixos-system-x86_64-linux.tar.xz.
  • For running the script, you’ll need to do sudo ./kexec_nixos instead of using the ubuntu user.
  • While mounting, mkdir /mnt should error because the directory already exists on Ubuntu.

Everything else worked without a hitch!

2 Likes

@noah Thanks for these suggestions. I needed to use all of them as well.

@korfuri Thanks for this write-up! I also successfully worked through it!

If I could make one suggestion on the article that @noah didn’t already hit, it is around the timing of the kexec and ssh thing. The article says:

Untar the resulting tarball and run kexec. This will take a while, but you’ll end up seeing a line that says + kexec -e . Yeah, it will look like it’s stuck for a good 5-10min, but be patient.

This explanation made me think that it would take 5 to 10 minutes for the + kexec -e line to be displayed, but after I saw that line then I could immediately re-ssh to the machine.

But when trying this, the + kexec -e line came up very quickly, but then it took about 5 to 10 min for the machine to come back up and be able to SSH into. That is to say, I was waiting for about 10 minutes after I saw the + kexec -e line until I was able to actually connect over SSH.

2 Likes

Thanks @noah for the feedback! I’ve updated the post accordingly.

@cdepillabout that’s interesting. I can’t run it right now but I remember there was a trap: it will show a similar line right away (iirc that would be + kexec -l), and then it takes a while to get to + kexec -e. Once kexec -e is invoked, the system is effectively booting stage 2 again, which may take some time depending on the config you have - I recommend including only the minimum in that config, since it’s temporary anyway, so basically enabling OpenSSH and setting an authorized_key and nothing more. That should boot pretty quickly, within seconds on that kind of machine shape.

2 Likes

Oh, maybe this was my mistake here! I think I only waited for a single + kexec line, and it was probably the -l instead of the -e.

I took some time to check the source and yeah that must be it. I edited the blog post again to make it more clear. Thanks!

1 Like

This method is mindblowing! :heart_eyes:

3 Likes

Played around with this server a bunch this weekend, and I found one weird quirk that seems to be particular to Oracle’s implementation.

I wasn’t able to reach the instance on ports that weren’t 22. For one thing, I had to make sure to open them in the VPC > Security List. But apparently (at least according to the Ubuntu solutions), you also need to set iptables by hand to get it to work.

So networking.firewall options like this don’t seem to open the ports:

$ sudo iptables -L INPUT --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    nixos-fw   all  --  anywhere             anywhere

But if you run sudo iptables -I INPUT -j ACCEPT, so it’s like this:

$ sudo iptables -L INPUT --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  anywhere             anywhere
2    nixos-fw   all  --  anywhere             anywhere

Then it works perfectly. Not sure if anyone knows a declarative solution for this, other than a systemd oneshot. But you’ll lose the ACCEPT rule after reboot.

NixOS firewall is enabled by default, so this make sense right?
You may prefer to open the ports through NixOS Search

1 Like

Yes, but I’m saying that the networking.firewall options don’t actually do anything here from my testing. It seems like Oracle is actually checking iptables for a particular pattern somehow.

Looking at pages like this, it seems that they force you to go to iptables directly on ubuntu instead of using ufw.

My theory is that because our VM started as ubuntu (or maybe just a quirk of OCI generally), it uses very particular iptables rules.

Maybe there’s a way around it? Or maybe I’m doing something wrong.

It’s unlikely the hypervisor is looking at the iptables rules within the kernel.

1 Like

Good point. You know, I’m looking at this again and I must have set the allowedTCPPorts for a particular interface instead of doing it for the firewall generally. Trying again with the correct settings seems to work well. Thanks!

But I will say, for those who are having trouble with hitting your Oracle server more generally, it’s probably because the VPC default security list is set to only open 22 at first. That also tripped me up for a while.

1 Like

In case you have a debit card instead of a credit card you might need to check for a cheap credit card first (cloud resources as described above sadly are for credit card owners only for now).

I can’t seem to be able to connect on the latest Ubuntu version after calling kexec (Note that I’ve waited for the kexec -e line to appear). The ssh connection times out and nothing appears when using oracle console connection. Rebooting does roll back to Ubuntu as expected.