Docker - switch // overlay / overlay2 FS lead to emergency console

Help
1

Hi,
i’m pretty new to nixos and made my first steps.
I’d installed nixos and wanted to enable docker;

  virtualisation.docker.enable = true;
  #virtualisation.docker.storageDriver = "overlay2";

which leaded after reboot to this error:
overlayfs lower directory not found.

i believe and maybe someone can assist me - that a nixos-generate-config creates the following entry which is not intended by the docker daemon

#  fileSystems."/var/lib/docker/overlay2/2759d93183010abe2e39e0516af98bb1d5cb5bdef11dc4460170c096d48e5873/merged" =
#    { device = "overlay";
#      fsType = "overlay";
#    };

which leaded to a fstab entry and then finaly into emergency console.

By deactivating this overlayfs entry in hardware-config the boot does work as expected.
Is this possible a side-effect of the generation - and should there be a warning in combination with docker? maybe ignore overlay-fs entry in a whole? a bug?

thanks for sharing your wisdom with me
Johannes

2

This post saved my day today. Unfortunately I also do now know how the entry in the hardware-configuration got there but I at least removing it got my system running again.

I would also gues it is a bug though. Maybe running nixos generate-config adds this entry from a running container instance which leads to the error when on boot nixos tries to mount a location that does not exist anymore?

Will investigate. This is the first time I experienced this error even though I have been using NixOS and docker for a while. However I only have today with this machine left and I am pretty happy that it is running again for now. Will keep you updated

3

I have experienced something similar. I have number of docker containers running using docker-compose. After adding another drive to my machine I ran nixos-generate-config and found that it added half a dozen entries corresponding to different docker containers. This definitely seems like undesirable behaviour…

4

I have the same problem since some time. Anybody found some work around?

5

Apparently this is happening when we make a switch rebuild with running containers.

The solution would be to stop the containers before switching, using docker stop $(docker ps -q)
Additionally, best practice when switching off computer, use sudo shutdown -r now instead of sudo reboot to avoid corrupt any volume.

In case you are embedding your commands to upgrade or install the system in a shell script, you might find useful a kind of hook, so before switching nix, check if containers are running.
You can find the script to handle docker in my repo
I call the script from install.sh or upgrade.sh
And I only update the system using these scripts so I have more control and don’t forget any checkup