JupyterHub database migration

nobodyinperson · June 4, 2025, 9:22pm

I am struggling hard to get my JupyterHub working again. After an update (longer ago), JupyterHub often requires a manual database migration and refuses to start:

❯ journalctl -efu jupyterhub
...
Jun 04 22:10:41 yann-desktop-nixos jupyterhub[1038575]: Found database schema version 0eee8c825d24 != 4621fec11365. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema.
...
Jun 04 22:10:42 yann-desktop-nixos systemd[1]: Failed to start Jupyterhub development server.

So far so good, no problemo, on other distros it’s literally just sudo jupyterhub upgrade-db --db=/var/lib/jupyterhub/jupyterhub.sqlite and off you go. But of course not on NixOS, where Python has to be a massive pain

Join me on a typical journey of NixOS pain

❯ nix run nixpkgs#python3Packages.jupyterhub -- upgrade-db --db /var/lib/jupyterhub/jupyterhub.sqlite
# ...
#    PermissionError: [Errno 13] Permission denied: '/var/lib/jupyterhub/jupyterhub.sqlite.2025-06-04-230501'

Right, owned by root, I’ll just sudo it:

❯ sudo nix run nixpkgs#python3Packages.jupyterhub -- upgrade-db --db /var/lib/jupyterhub/jupyterhub.sqlite
error: flake 'flake:nixpkgs' does not provide attribute 'apps.x86_64-linux.python3Packages.jupyterhub', 'packages.x86_64-linux.python3Packages.jupyterhub', 'legacyPackages.x86_64-linux.python3Packages.jupyterhub' or 'python3Packages.jupyterhub'

… Apparently root’s registry needs to be handled manually (why!?), so we do that first:

❯ sudo nix registry pin nixpkgs github:NixOS/nixpkgs/nixos-25.05

Surely, now it’ll work?

❯ sudo nix run nixpkgs#python3Packages.jupyterhub -- upgrade-db --db /var/lib/jupyterhub/jupyterhub.sqlite
...
ModuleNotFoundError: No module named 'jupyterhub'
...
    subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpvpu5qs5i/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 1.

Full error

❯ sudo nix run nixpkgs#python3Packages.jupyterhub -- upgrade-db --db /var/lib/jupyterhub/jupyterhub.sqlite
[sudo] Passwort für yann: 
[I 2025-06-04 23:23:10.248 JupyterHub dbutil:129] Upgrading sqlite:////var/lib/jupyterhub/jupyterhub.sqlite
[I 2025-06-04 23:23:10.248 JupyterHub dbutil:100] Backing up /var/lib/jupyterhub/jupyterhub.sqlite => /var/lib/jupyterhub/jupyterhub.sqlite.2025-06-04-232310
Traceback (most recent call last):
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/bin/.alembic-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/config.py", line 636, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/config.py", line 626, in main
    self.run_cmd(cfg, options)
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/config.py", line 603, in run_cmd
    fn(
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/command.py", line 408, in upgrade
    script.run_env()
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/svl690bn2mdg1q5va9g9cddn8xww6gnl-python3.12-alembic-1.15.2/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/nix/store/8iv8w6j6rhfwhp22fgawv17i2g7rr9ww-python3.12-jupyterhub-5.3.0/lib/python3.12/site-packages/jupyterhub/alembic/env.py", line 32, in <module>
    fileConfig(config.config_file_name)
  File "/nix/store/8w718rm43x7z73xhw9d6vh8s4snrq67h-python3-3.12.10/lib/python3.12/logging/config.py", line 83, in fileConfig
    formatters = _create_formatters(cp)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/8w718rm43x7z73xhw9d6vh8s4snrq67h-python3-3.12.10/lib/python3.12/logging/config.py", line 132, in _create_formatters
    c = _resolve(class_name)
        ^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/8w718rm43x7z73xhw9d6vh8s4snrq67h-python3-3.12.10/lib/python3.12/logging/config.py", line 101, in _resolve
    found = __import__(used)
            ^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'jupyterhub'
[E 2025-06-04 23:23:10.576 JupyterHub app:3920]
    Traceback (most recent call last):
      File "/nix/store/8iv8w6j6rhfwhp22fgawv17i2g7rr9ww-python3.12-jupyterhub-5.3.0/lib/python3.12/site-packages/jupyterhub/app.py", line 3918, in launch_instance_async
        await self.start()
      File "/nix/store/8iv8w6j6rhfwhp22fgawv17i2g7rr9ww-python3.12-jupyterhub-5.3.0/lib/python3.12/site-packages/jupyterhub/app.py", line 3675, in start
        self.subapp.start()
      File "/nix/store/8iv8w6j6rhfwhp22fgawv17i2g7rr9ww-python3.12-jupyterhub-5.3.0/lib/python3.12/site-packages/jupyterhub/app.py", line 241, in start
        dbutil.upgrade_if_needed(hub.db_url, log=self.log)
      File "/nix/store/8iv8w6j6rhfwhp22fgawv17i2g7rr9ww-python3.12-jupyterhub-5.3.0/lib/python3.12/site-packages/jupyterhub/dbutil.py", line 134, in upgrade_if_needed
        upgrade(db_url)
      File "/nix/store/8iv8w6j6rhfwhp22fgawv17i2g7rr9ww-python3.12-jupyterhub-5.3.0/lib/python3.12/site-packages/jupyterhub/dbutil.py", line 85, in upgrade
        check_call(['alembic', '-c', alembic_ini, 'upgrade', revision])
      File "/nix/store/8w718rm43x7z73xhw9d6vh8s4snrq67h-python3-3.12.10/lib/python3.12/subprocess.py", line 413, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['alembic', '-c', '/tmp/tmpw2j4gw82/alembic.ini', 'upgrade', 'head']' returned non-zero exit status 1.

what’s this alembic thing now? Apparently a database migration tool or something. Apparently somewhere down the line, jupyterhub upgrade-db calls this alembic executable, and this alembic then itself wants to import the jupyterhub module, which it for some reason doesn’t see

Surely, a shell with both will work:

❯ sudo nix shell nixpkgs#python3Packages.jupyterhub nixpkgs#python3Packages.alembic --command jupyterhub upgrade-db --db=/var/lib/jupyterhub/jupyterhub.sqlite
(spoiler alert: no 😑)

Surely, more wizardry will solve it.

❯ sudo nix shell --impure --expr 'with import <nixpkgs> {}; python3.withPackages (ps: [ ps.jupyterhub ps.alembic ])' --command jupyterhub upgrade-db --db=/var/lib/jupyterhub/jupyterhub.sqlite
(spoiler alert: no 😑)

Now what…

It feels like this alembic needs to be wrapped to somehow “see” the jupyterhub module…

I find it very curious that I can’t find anything on this topic. Nothing about this error message. Apparently nobody uses services.jupyterhub.enable=true. Or everybody just gave up

nobodyinperson · June 4, 2025, 9:38pm

Aha! There is something to be found on this topic, even a PR that would automate the migrations, interesting:

github.com/NixOS/nixpkgs

nixos/jupyterhub: add automatic migrations

master ← jluttine:jupyterhub-automatic-migrations

opened 11:43AM - 08 Sep 23 UTC

jluttine

+11 -0

## Description of changes Sometimes when JupyterHub is upgraded, migrations n…eed to be run on the database: https://jupyterhub.readthedocs.io/en/stable/howto/upgrading.html#upgrade-jupyterhub-database Currently, JupyterHub service on NixOS, doesn't run these automatically, so after upgrading the service stops working with an error like this: ``` Sep 07 00:16:27 jupyter jupyterhub[2281780]: Found database schema version 4dc2d5a8c53c != 0eee8c825d24. Backup your database and run `jupyterhub upgrade-db` to upgrade to the latest schema. Sep 07 00:16:27 jupyter systemd[1]: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE Sep 07 00:16:27 jupyter systemd[1]: jupyterhub.service: Failed with result 'exit-code'. ``` One would need to run the following command: ``` jupyterhub upgrade-db ``` But this is surprisingly tricky given all the nix-related properties regarding separate environments: One needs to figure out the path to the correct executable `jupyterhub` and its config file. One can read these from the output of `systemctl status jupyterhub` and then manually write something as follows: ``` /nix/store/.../bin/jupyterhub upgrade-db --config /nix/store/...-jupyterhub_config.py ``` And actually this didn't work for me. I had to even define `PYTHONPATH` because JupyterHub launches alembic as a subprocess and for some reason it doesn't find JupyterHub when trying to import it. So, I had to run: ``` PYTHONPATH=/nix/store/.../lib/python3.10/site-packages /nix/store/.../bin/jupyterhub upgrade-db --config /nix/store/...-jupyterhub_config.py ``` So, instead of letting JupyterHub service to fail unnoticeably after upgrades and requiring users to figure out these complex manual commands, NixOS could just run the migrations automatically. This PR makes the JupyterHub service to run the database migrations automatically, but the user can opt-out. An alternative would be to create wrapper for the `jupyterhub` executable which sets all the paths and configs correctly, and the user could just run something like `manage-jupyterhub upgrade-db`. Of course, these to solutions don't need to be mutually exclusive, but I decided to implement only the automatic migrations for now. Things to do still: - [ ] I'm worried the current implementation doesn't work without setting `PYTHONPATH`. I don't understand why it's needed though. To me it sounds like there's a bug in the wrapping of `jupyterhub` or `alembic` executables, but probably it's all good and I just don't know how to handle this. If setting `PYTHONPATH` is the correct solution, perhaps it should be done on the service environment level. - [ ] I'd like to add NixOS tests about this feature. ## Things done - Built on platform(s) - [ ] x86_64-linux - [ ] aarch64-linux - [ ] x86_64-darwin - [ ] aarch64-darwin - [ ] For non-Linux: Is `sandbox = true` set in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html)) - [ ] Tested, as applicable: - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests)) - and/or [package tests](https://nixos.org/manual/nixpkgs/unstable/#sec-package-tests) - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test) - made sure NixOS tests are [linked](https://nixos.org/manual/nixpkgs/unstable/#ssec-nixos-tests-linking) to the relevant packages - [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage) - [ ] Tested basic functionality of all binary files (usually in `./result/bin/`) - [23.11 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2311.section.md) (or backporting [23.05 Release notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2305.section.md)) - [ ] (Package updates) Added a release notes entry if the change is major or breaking - [ ] (Module updates) Added a release notes entry if the change is significant - [ ] (Module addition) Added a release notes entry if adding a new NixOS module - [ ] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).