When should I change system.stateVersion?

Hi,

The comment on top of system.stateVersion in the default configuration.nix says:

  # This value determines the NixOS release with which your system is to be
  # compatible, in order to avoid breaking some software such as database
  # servers. You should change this only after NixOS release notes say you
  # should.

However, I did not find a single mention of stateVersion in the release notes in the Nixos manual. Am I ever supposed to change this?

2 Likes

Relevant discussion:
https://github.com/NixOS/nixpkgs/pull/50112

1 Like

This does seem to ask the same question:

2 Likes

Hi,
it may be a wrong practice however I have few NixOS systems installed
since 17.03, regularly upgraded in place changing state version after
the upgrade. They still run issueless.

Of course sooner or later I may face problems, and my small installed
base is certainly not a statistic sample, however that’s the result
I find…

– Ingmar

Most of the time you won’t actually care about stateVersion.

It’s generating issues with postgresql in particular. Typically,
stateVersion = "18.03"; and stateVersion = "18.09"; are exactly
the same thing, as there has been no backwards-compatibility breakage.

The idea is: a bump of stateVersion potentially can’t be reverted, and
potentially needs manual intervention (eg. with pgsql upgrades).

See also Renames stateVersion to stateEpoch by samueldr · Pull Request #50112 · NixOS/nixpkgs · GitHub that proposes to
reduce confusion about stateVersion :slight_smile:

1 Like

It’s really easy to grep the nixpkgs repo for instances of stateVersion. There are so few that it is really easy to audit if updating it will affect you.

2 Likes

Maybe I’m not able to put the pieces together, but I don’t understand how the replies in this thread, or the proposed change PR, answer the questions posed by the OP. For example, I’ve read them multiple times, on different days, and still don’t understand the answers to the following questions:

  1. Do I even want to try to update stateVersion? I’m the type of person that sees an old date, assumes that means something is old or stale and that I probably want to “upgrade”. (I do understand enough to know that the date is not signifying the release I’m on, I’m on my own nixpkgs branch anyway. And thus far have not upgraded modified stateVersion on any of my old systems)

  2. If I do want to update it, what is the procedure for doing so? Is it just a matter of eliminating/resolving any deprecation/rename notices? Does that guarantee I get everything? Or do I need to manually grep the code for the stateVersions between “then” and “now” to see all changes explicitly? (Another way of asking this, can state changes be made without adding a warning/assert?)

  3. How does all of this interact with my nix configuration that is checked into my repository? What happens when my nixcfg has “stateVersion = 18.09” in it, but I deploy it with a “20.09” ISO? How can it possibly know the right thing to do? Does nixos-install protest if the initial stateVersion differs from the version being installed?

9 Likes

Cole Mickens nixos1@discoursemail.com writes:

Maybe I’m not able to put the pieces together, but I don’t understand how the replies in this thread, or the proposed change PR, answer the questions posed by the OP. For example, I’ve read them multiple times, on different days, and still don’t understand the answers to the following questions:

  1. Do I even want to try to update stateVersion? I’m the type of person that sees an old date, assumes that means something is old or stale and that I probably want to “upgrade”. (I do understand enough to know that the date is not signifying the release I’m on, I’m on my own nixpkgs branch anyway. And thus far have not upgraded modified stateVersion on any of my old systems)

If you want to get something that’s fenced on a newer stateVersion than
you have, yes. Otherwise, you may want to do it from time to time still
to avoid having debian-age versions of packages lying around your
dependencies.

The point of the stateEpoch proposed change is to reduce that feeling of
seeing an old date by making it opaque :slight_smile:

  1. If I do want to update it, what is the procedure for doing so? Is it just a matter of eliminating/resolving any deprecation/rename notices? Does that guarantee I get everything? Or do I need to manually grep the code for the stateVersions between “then” and “now” to see all changes explicitly? (Another way of asking this, can state changes be made without adding a warning/assert?)

Grepping the code for stateVersion, even though theoretically everything
should be pointed out in release notes. State changes only appear with a
test against stateVersion, but won’t cause warnings.

  1. How does all of this interact with my nix configuration that is checked into my repository? What happens when my nixcfg has “stateVersion = 18.09” in it, but I deploy it with a “20.09” ISO? How can it possibly know the right thing to do? Does nixos-install protest if the initial stateVersion differs from the version being installed?

No, it’ll build the configuration for a 18.09 state independently of the
recent-ness of the ISO. Unless support for older states is removed, but
we still have 14.12 lying around, so you can likely assume that won’t
happen often :slight_smile:

3 Likes

Hi,

Most of the time you won’t actually care about stateVersion.

It’s generating issues with postgresql in particular. Typically,
stateVersion = "18.03"; and stateVersion = "18.09"; are exactly
the same thing, as there has been no backwards-compatibility breakage.

The idea is: a bump of stateVersion potentially can’t be reverted,
and potentially needs manual intervention (eg. with pgsql upgrades).

sorry for late answer and thanks for the insights :slight_smile:

Honestly the first time I upgrade NixOS was after few days after the
first NixOS install/usage for me so I do not really read enough docs,
simply trying to see if NixOS is something good/interest for me…

After I see no problems in upgrading state version so I keep going,
using zfs before and nilfs2 after I can “revert” the hard way so I’m
not really afraid of breaking things.

Can I “quickly see” differences between state versions so to being
able to decide, with knowledge, if something is change for any single
deploy, so to decide change or not change? I suppose in the long terms
I will not been able to upgrade a system with too ancient state
version (something like after 10/20 NixOS releases for instance).

– Ingmar

Hello! I have actually bumped into this problem once. Here is article from Wiki: FAQ/When do I update stateVersion - NixOS Wiki

The “stateVersion” manages only deep system-relevant things that are very few and have no impact on actual system. There are work-arounds for every new release that compare the state variable and do the few actions differently, based upon value of that variable.
You can upgrade it, but you shouldn’t. Because: 1) it will add no benefit at all from user standpoint, it will apply a different (more recent) structure of metadata 2) will require you to implement the changes or risk breaking the system because it looses atomic state and you can get very strange problems with packages. Which is what I did and got.

I recommend to leave it alone and never change. Consider full reinstall instead. Once in 10 years or so?

Now, when I had this same question, I got an interesting advice from user infinisil over IRC, which explains it very good:

< infinisil > :
stateVersion is used by nixos for state whose default changed throughout different versions, e.g. a database directory now being in /var/db instead of /var/lib/db.
So there is a switch: dbDir = if stateVersion > 18.03 then /var/db else /var/lib/db.

Updating this variable would mean that suddenly it thinks the db is in a different place, so it breaks.
You can update it if you really want it, but then you need to go through the list of changes and manually migrate everything.

That means moving the database from /var/db to /var/lib/db yourself.
Just don’t update it, it’s not worth it.

The only thing it gets you is that the string “17.09” isn’t in your config anymore and potential problems, literally nothing else

8 Likes

Hi,

Thank you all for your helpful responses. I understand now that I should never want to change stateVersion. I think the biggest issue here is that the comment which has the mission of warning users not to touch the value, gives the impression that the user should change the value from time to time. User is unintentionally lead to think that this would be a regular housekeeping operation in nixos. I suggest that it be reworded as “you shouldn’t need to change this ever” or something similar.

9 Likes

I opened a pull request to make the comment on default configuration.nix better. If you have any ideas how to reword it, I would appreciate your input:

https://github.com/NixOS/nixpkgs/pull/74138

Sorry to revive this topic, but when I read all the answers, I was thinking that stateVersion is very minor and is just used to make sure that, basically, “path” matches, and that having an old number is not a problem at all:

However, I think that the above statement is confusing. Indeed, I just checked on my system and in the nix code (not so hard to read for that, just grep stateVersion), and keeping an old version number will actually have an impact on the system, since it will use older packages.

For example after an upgrade on which I forgot to specify system.stateVersion, my system was using postgresql 11.1 (and it also lost the old database since it uses the path /var/lib/postgresql/11.1 to store the database). But when I realized that my webserver was unable to access the database, I added:

system.stateVersion = "18.09";

and then the version of postgresql used by the system was suddently 9.6 (and it also changed the path of the database to /var/lib/postgresql/9.6, so my database came back to life !). So if people care about having recent software, I think that it does make sense to update the stateVersion from time to time, but the user needs to make sure to apply manually all the changes given in the release, or/and be prepared to breakages.

I just created a new post here for people interested by an automatic script.

1 Like

There is no need to continue after this statement. To explain, first and last time I had bumped this attribute, my whole system was left in inconsistent state without ability to revert to the previous generation. The effect was not immediately obvious, but then on next rebuild, parts of the system started to give warnings, then unrelated packages refused to build spitting strange errors. I have spent about a day trying to figure out the reason in nixos IRC with other members shaking their heads, until Infinisil directly asked if I have touched this attribute in the past. After complete wipe and rebuild using the very same configuration.nix without any changes all the errors and problems magically disappeared. No, user should not touch this attribute at all. Manually applying changes in this area goes against NixOS policy itself of a functional, consistent, stateless environment and will introduce deep breakages in the nix own toolchain. This attribute affects only internal agreements deep inside the base system and adapts the new changes in newest revisions automatically in order to maintain the mentioned policy. In the moment this attribute is manually changed, system goes into untested inconsistent state with no guarantee for anything.

I directly state that this value should not be touched in any case, unless full physical backups are done and user understands the risks of getting system irreversibly broken and/or the breakage is the actual goal.

The only reliable way to bump this attribute seems to create a partitioning scheme, similar to some Android devices with two root partitions A & B, only one of which is marked active A. The configuration.nix should then be transfered to a different partition, chrooted, attribute value bumped and toolchain should build a new, disconnected system on a passive B paritition. Then boot sequence is changed in a way that active and passive are flipped and machine is rebooted into new system B (not just generation), from the point of which user can decide to wipe the old ex-active partition or clone it or keep it or decide to do anything similar.

I think it has never been present on my system before, not sure if it is (was?) a bug of the raspberry image. But now I put one!

Really? When you look at the code I can’t see many such fundamental operation impossible to apply manually, I mostly see path updates and sofware versions update. I’m quite curious actually, do you have an example? Or maybe manually moving the path can be a problem if some software put hard links to the old folder? Then maybe putting a symlink could avoid the problem? Any other issue could appear?

So then what is the way to go if I don’t want to live with old and unmaintained packages? Is it safer for example to overwrite the attributes like services.postgresql.package once I manually migrated postgresql? Because if it’s safe I don’t really see the difference with doing that thing for all packages and then changing stateVersion, and if it’s not safe either… then I guess it means that the user needs to live with outdated software, which is not really a great idea…

1 Like

The problem is that touching this attribute will modify a running horse. It is an issue, its a compromise that exists in order to achive stateless state. I am not happy with it at all, I am not objecting work towards making migration to newer state possible - I merely state that in its current form its modification will corrupt the system and it stretches to far more than just a db upgrade.

Perhaps we should have nixos-rebuild yell at the user if it detects a change to this attribute, and require a special flag in order to proceed with the rebuild.

It might be worth noting, that the cited comment from the configuration.nix is now out-dated.

https://github.com/NixOS/nixpkgs/pull/77279

On the other hand I had only made that commit with the “better than nothing” mentality and it might be worth to improving on this further.

1 Like

Thank you for taking your time and contributing a better description! This will definitely prevent a dozen (or more) totally broken systems in a weird and unpredictable way.

2 Likes

Recent attempt to upgrade a nixos server I started in 2018 to nixos-22.11-small generated this warning:

trace: warning: Enabling both boot.enableContainers & virtualisation.containers on system.stateVersion < 22.05 is unsupported.

So now it seems stateVersion is tied to the operation of new features. So if no user should touch this attribute, is this a recommendation to scrap the server and start over from scratch?

3 Likes