/dev/null: Permission denied?

Help
1

G’day,

I hope you are doing well.

I’ve encountered something strange on 24.05, and I don’t know if I’ve done something silly, or what is causing this. I’m wondering if anyone else has seen this?

When I try to tab complete, or scp files to the machine I see permission denied errors.

[das@hp4:~]$ ls -bash: /dev/null: Permission denied
bash_completion: _comp_upvars: `-a2': invalid number specifier
-bash: /dev/null: Permission denied
bash_completion: _comp_upvars: `-a0': invalid number specifier
-bash: /dev/null: Permission denied
bash_completion: _comp_upvars: `-a2': invalid number specifier
-bash: /dev/null: Permission denied
bash_completion: _comp_upvars: `-a0': invalid number specifier

das@t:~$ ssh hp4
Last login: Thu Jun 20 15:57:43 2024 from 172.16.50.197
-bash: /dev/null: Permission denied

[das@hp4:~]$

So somehow the permissions have changed on /dev/null

[das@hp4:~]$ ls -la /dev/null
crw-r--r-- 1 root root 1, 3 Jun 18 17:03 /dev/null

My fix is “sudo chmod 777 /dev/null”. This usually works for a few minutes, and then it can happen again. I’m not sure what could possibly be changing this, nor am I sure how to debug this. Maybe I could setup something like an inotify watch?

das@t:~$ ssh hp4
Last login: Thu Jun 20 15:57:43 2024 from 172.16.50.197
-bash: /dev/null: Permission denied

[das@hp4:~]$ sudo chmod 777 /dev/null
[sudo] password for das: 

[das@hp4:~]$ exit
logout
Shared connection to hp4 closed.
das@t:~$ ssh hp4
Last login: Thu Jun 20 15:57:48 2024 from 172.16.50.197    <--- error has gone

[das@hp4:~]$

Thanks in advance,
Dave

2

It’s probably unrelated, but I’ve seen something similar: it’s just that instead of /dev/null having wrong permissions, the file was being removed altogether. When that happens all hell breaks loose, because the first script that does >/dev/null as root will recreate it as a normal file with wrong permissions. (besides writing huge amounts of garbage to the disk that should have been discarded).

For reference, if you want to recreate /dev/null, here’s a command:

sudo sh -c 'rm /dev/null; mknod /dev/null c 1 3; chmod 777 /dev/null'

I haven’t figured out yet who is deleting /dev/null, and it’s been happening for a couple of years now, thankfully not that often… I should try to write some eBPF filter or something similar to catch the process in the act.

4 Likes
3

Thanks for the reply! Curious indeed. It almost feels like systemd (or similar) could supply a virtual /dev/null that was unique for every process…

4

Setting the immutable bit should probably be enough to prevent random processes from messing with it.

5

Good idea, but it looks like chattr doesn’t let you set immutable.

[das@hp4:~]$ sudo sh -c 'rm /dev/null; mknod /dev/null c 1 3; chmod 777 /dev/null; chattr +i /dev/null'
[sudo] password for das: 
chattr: Operation not supported while reading flags on /dev/null
6

Another, more involved, idea is to set up a VM with your config and bisect that, slowly adding (or removing) options from the VMs copy your config until you find out what causes this.

7

It’s unfeasible in my case: it happens far too rarely to be able to bisect in a reasonable timespan. I came up with this trap:

bpftrace -e 'tracepoint:syscalls:sys_enter_unlinkat { if (strcontains(str(args.pathname), "null")) { printf("%d, %s: %s\n", pid, comm, str(args.pathname)); } }'

Assuming /dev/null will be deleted with unlink it should tell me who did it.
Now I only have to wait for the process to take the bait.

2 Likes
8

Uhm…

39761, nix-collect-gar: cgywj248r52nbkb4kk5yhbw9s6lqpj2i-crate-windows_i686_gnullvm-0.5..
756, git: src/nixpkgs/pkgs/by-name/nu/nulloy/package.nix
41898, nix-daemon: null