Writable nix store

i just made a writable-nix-store.js script
to get temporary write-access to existing files in my /nix/store

it’s useful for quick debugging and prototyping of script files
(python, bash, cmake, node, perl, …)
faster than patching the source files and building new derivations

this works by

  1. bind-mounting the original store to /a/nix/store
  2. mounting an overlayfs on /nix/store with the upper dir /b/nix/store

start overlay

sudo writable-nix-store.js start
success: mounted overlayfs:
  overlay.hide:  /nix/store
  overlay.mount: /nix/store
  overlay.work:  /nix/overlay-store/work
  overlay.lower: /a/nix/store
  overlay.upper: /b/nix/store

now all modified (or new) files go to /b/nix/store, but not to /nix/store

edit files

for example, let’s edit python’s http server module

server_py=$(dirname $(python -c "import http; print(http.__file__)"))/server.py

echo $server_py
# /nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/http/server.py

sudo sed -i 's/import argparse/print("hello"); sys.exit()/' "$server_py"

find /b -type f
# /b/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/http/server.py

python3 -m http.server 
# hello

#diff -u /a/$server_py /b/$server_py
diff -u /{a,b}/$server_py
--- /a/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/http/server.py	1970-01-01 01:00:01.000000000 +0100
+++ /b/nix/store/hym1n0ygqp9wcm7pxn4sfrql3fg7xa09-python3-3.9.12/lib/python3.9/http/server.py	2022-05-03 17:03:28.368480361 +0200
@@ -1253,7 +1253,7 @@
             sys.exit(0)
 
 if __name__ == '__main__':
-    import argparse
+    print("hello"); sys.exit()
     import contextlib
 
     parser = argparse.ArgumentParser()

restore files

sudo cp /a/$server_py $server_py

python3 -m http.server
# Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

stop overlay

note: all new files in /nix/store since start are now in /b/nix/store

sudo writable-nix-store.js stop

happy patching : )

4 Likes

You should just use nix --store ./myStore which will give you the ability to make a nix store anywhere. Actually modifying the contents of the nix store will result in near certain corruption and death though, so be careful.

user: matthew ~ 
❯ cd tmp

user: matthew ~/tmp
❯ nix build nixpkgs#hello --store ./myStore

user: matthew ~/tmp
❯ ls -lah ./myStore
total 37K
drwxr-xr-x   3 matthew users   3 May  3 19:23 .
drwxr-xr-x 164 matthew users 310 May  3 19:23 ..
drwxr-xr-x   4 matthew users   4 May  3 19:23 nix

user: matthew ~/tmp
❯ ls -lah ./myStore/nix/
total 67K
drwxr-xr-x 4 matthew users   4 May  3 19:23 .
drwxr-xr-x 3 matthew users   3 May  3 19:23 ..
drwxr-xr-x 8 matthew users 293 May  3 19:23 store
drwxr-xr-x 3 matthew users   3 May  3 19:23 var

user: matthew ~/tmp
❯ touch ./myStore/nix/store/foo
1 Like

scary ^^
the actual nix store stays read-only, and only the overlay’s upper dir is writable
worst case: i modify or delete some critical file, and my system hangs
then i just need to reboot, to restore my original nix store

looks good on paper, but im afraid that will “install the world” into the new store
i want a quick solution, so i simply mutate the existing files

2 Likes

how to break your nixos with this tool:

  • start overlay
  • change /etc/nixos/configuration.nix
  • run nixos-rebuild switch
  • stop the overlay

now many programs are broken, because the symlink-targets have moved from /nix/store to /b/nix/store

in the worst case, you can’t run sudo and you need a hard reboot.
then boot a previous generation of your nixos config.
nix-build will throw
error: getting status of '/nix/store/*': No such file or directory
fix: run nix-store --verify --repair
to sync the database /nix/var/nix/db/db.sqlite with the files in /nix/store

i have fixed my tool, to disable nixos-rebuild while the overlay is active
so “breaking your nixos” should not be possible

todo: rewrite in bash

todo: also mount an overlay for /nix/var/nix/db/db.sqlite

todo: use lockfiles of nix

1 Like

Why not make use of user namespaces to create a container which has the overlay filesystem, while the rest of the system still sees the correct, immutable nix-store? If you never ever ever run nix in any way inside the container, this should actually be safe, and it’s that much easier to get out of it if you screw up.

1 Like

overlayfs was the simplest solution and im happy with it

i want to make this a standalone tool with a bit more polish, because it makes prototyping of patches so much faster (“make nix more user-friendly”)

im happy to integrate alternative backends, so feel free to experiment (ideally in bash, but javascript/python is fine too)