Help. I was configurating my nix binary cache mirror. The system will freeze permanently ranging from 20 seconds to 2 minutes after booting into the system even in tty1. Thus rendering the whole system unusable.
I got the chance to capture some photos before restarting.
I am so frustrated because I heard that this system was suppose to be stable and hard to break.
Now it can’t even function normally.
Hello, that’s not really KDE crashing but the amdgpu driver for your graphics card.
Did the crashes start following a kernel update? If yes, you wanna roll that back (i.e. boot from an older, know to work, revision of your system).
Thanks for letting me know that it is the kernel’s problem.
It has been happening for a while until I stopped using a third party theme. It started working fine but it just happened recently after I did garbage collection.
It hasn’t been stable ever since I started using nixOS. I have just started using nix os so I definitely have lots of problems with my config file.
I am working on changing my binary cache mirror to a closer one so I can properly update my system to see if I can fix it.
BTW, could the program crashing be related to the kernel too?
Am I the only one experiencing such problem?
If your program crashes at the same time that this error messages shows up in the kernel log with at drivers/gpu/drm/amd/amdpu (you can monitor the kernel logs with dmesg -w, then you can assume that they are related.
The AMD GPU drivers are not very stable, and that’s an euphemism, e.g. in issue #3579 people had crashes (forcing you to reboot) every other day for almost year before reporting the issue, because we are all so used to regressions being fixed in a kernel update. Looks like this one might never get fixed…
You are not the only one having problems like that.
Thank you so much for your reply.
I have got the following questions:
If the driver is bad, which kernel should I rollback to?
How can I specify it in configuration.nix?
Why is the amd gpu driver built in the kernel? Can I use amd radeon’s official driver instead?
I was using linux mint with nvidia igpu on a laptop before and it never crashed ONCE. Why are recent kernels causing the most problem?
Does Wayland has something to do with the crashes?
We need to know when your computer started to crash, and whether that followed a system update.
Older kernel versions are packaged and can be used as documented.
I was under the impression that the amd gpu driver built in the kernel is the official one.
Nvidia and AMD drivers are completely different. Assuming this is a driver issue (as opposed to e.g. a hardware issue), you can search for your issue or open a new one at: Issues · drm / amd · GitLab.
If Wayland has something to do with it (or anything else really), it should result in the driver spitting out an error, not crashing.
I have done dmesg -w and it says the following:
I have also done journalctl and it says the following:
Ok, those errors look a little different now, and smell like a hardware issue, can you reboot into Memtest86+, and see if you can get few passes error free?
I just built this pc last month and has dual ssds, one for windows, one for nixos. Windows ran with 0 issues so I believe it is 100% software issue.
Well, software issue is a given. You fucked with your nix pkg cache, its likely trying to pull bad pkg data and getting borked.
You probably should drop your current nix-channels, re-add the stable and nixos channels, and run a full rebuild
EDIT: Actually had to this recently for something similar, let me find the right nix channels for current stable and ill post a potential solution
Thanks for the test, and congrats on the new computer.
Something is definitely up with your install, it might have been / is caused by some hardware issue, it’s hard to say.
Since the computer is new, I imagine you have little data, can you save your NixOS configuration, flash a thumbdrive with a fresh ISO, and go through a reinstall? You could keep your existing partitions and reformat them (maybe except your /home partition if you have one). If you want to change your partition layout, you might find disko helpful to write it down and apply it.
My intention is to start fresh, and overwrite any kind corruption that could have happened at the filesystem level. It should be easy to re-apply your NixOS configuration after that. Make sure you save whatever modifications you have in /etc/nix/nix.conf as well, you could re-use that file in the installer (don’t forget to systemctl restart nix-daemon after changing it), since I know cache.nixos.org is awfully slow where you are, and you need to use some local mirror.
You might find Ventoy helpful here, leave some space at the end (see reserved disk space section in in this page), and make a partition there to put your configuration.nix and nix.conf, that you could mount and access from the installer.
Last but not least, you shouldn’t see any of those segfaults if you run dmesg in the installer as well.
This is a less aggressive option than my above post. Pick the approach your prefer.
Much so, and def the first thing we should check. Integrity of the nixpkgs channel and its cache can cause some gnarly errors.
Get into root shell. IMPORTANT:MUST BE ROOT
sudo -s
nix-channel --list
It should return 2 links, you can remove them with their alias or their link. Use the alias as it should still be the default on your machine.
nix-channel --remove nixpkgs
nix-channel --remove nixos
now we add the channels that should be present back
nix-channel --add https://github.com/NixOS/nixpkgs/archive/release-24.11.tar.gz nixpkgs
nix-channel --add https://nixos.org/channels/nixos-24.11 nixos
Now we want to do a pull from these channels and then a rebuild
nix-channel --update
should return
unpacking 2 channels…
Good, thats what we need to see. After: While still root;
nixos-rebuild switch
Once the rebuild finishes, we need to do a full reboot to ensure all the new variables are set correctly and propagated to the entire system
also as root:
poweroff -r now
Test agan.
Side note:
If you are looking to dev your own pkgs and mess with source code and what not, i solved this by forking the nixpkgs repo from github and adding that new repo as a source for nix-channel nixpkgs.
This way you have much tighter control over what actually gets written to the pkg cache, and if you bork it, you can simply roll back the changes to the nixpkgs channel and rebuild.
Thanks everyone for your reply.
Please check out my previous post about changing my mirror.
https://discourse.nixos.org/t/help-i-broke-my-nix-channel/58309/8
Long story short, I currently live in China and the Chinese firewall will either block certain website or severely limit bandwidth to almost unusable, unfortunately cache.nixos.org is one of them.
You got a point, but I think it is the complete opposite, because I was using cache.nixos.org, every time I ran nixos-rebuild switch, there will be some http error and packages cannot be downloaded completely, I had to cancel the process many times while I was figuring out what is going on.
The incomplete downloads at the time definitely messed up parts of the system and the possible kernel amdgpu driver issue is adding fuel to the fire.
That is why I am desperately trying to use a Chinese mirror for my channel and binary substituter.
(
mirrors.tuna.tsinghua.edu.cn
mirror.sjtu.edu.cn
)
EDIT:
Thanks everyone for your suggestion but I am not rolling the system back because the current system is more stable than all the previous one. I fixed lots of issues and it will only get better. The current system is still usable and can be rescued from all your help and guidance.
I currently live in China, therefore the Chinese firewall will create unexpected issues. Sorry for the trouble.
I migrated from Linux Mint and I boasted lots about how advanced NixOS is, I would like to fix the problems and stay.
Nuff said.
In that case, i suggest trying to get a copy of your mirror prior to changes and run a diff on the 2 and see if theres something there to tell you where it got borked.
Is it possible to get a copy of the original mirror prior to changes you made?
Also, something to note, you mentioned using a custom pack for a theme and ive never had good luck with those on any DE. They either end up slowing the system as a whole down or borking a bunch of shit in the process too.
Your DE being smashed could actually produce the result you are currently seeing, if you’ve made any src changes to the DE, try rolling those back to default values, preferably some value prior to any custom mods so you have an A to B comparison
EDIT: Read your edit.
Yeah that makes alot of sense. I can see why you are struggling.
hmm…
EDIT 2:
Give me the mirror, i wanna try something on my own system and see if can reproduce your results
EDIT 3: Nevermind, found it in your other post, give me a few
What is the current output of
as root:
nix-channel --list
> nixos https://mirrors.tuna.tsinghua.edu.cn/nix-channels/nixos-24.11
Cool, dump the output of this file:
/etc/nix/nix.conf
I need to know a few things
# WARNING: this file is generated from the nix.* options in # your NixOS configuration, typically # /etc/nixos/configuration.nix. Do not edit it! allowed-users = * auto-optimise-store = false builders = cores = 0 max-jobs = auto require-sigs = true sandbox = true sandbox-fallback = false substituters = https://mirror.sjtu.edu.cn/nix-channels/store system-features = nixos-test benchmark big-parallel kvm trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= trusted-substituters = trusted-users = root extra-sandbox-paths =