Say there is a program which stores data in a user directory, and imagine it’s developers release a new version which is not compatible with the format of the old data - and thus require the user to perform a manual transition of the local data format with care (and probably backing up the old data). What would be the best way to introduce such a version bump in Nixpkgs?
I was thinking of this way (imagine the version bump is from 1.5 to 2.0):
Create a new attribute pkg2 that points to the new 2.0 version.
Add the attribute pkg1 that points to the old 1.5 version.
Make the old pkg attribute that used to be 1.5 point to pkg1, but use lib.warn in aliases.nix, with a message indicating that upstream introduced a breaking change and in order to avoid this evaluation warning the user should knowingly transition to pkg2 by themselves.
After a few months, remove the lib.warn, and make pkg = pkg2;.
There is also the option to start with pkg = lib.warn "a different message" pkg2; in aliases.nix, but that may be too risky depending on the behavior of the program with the old data.
2 packages I use have had such major version bumps that were added to Nixpkgs as separate attributes, and I had no way of knowing about them, unless I had checked myself at upstream’s releases, or in search.nixos.org or in Nixpkgs’ pull requests. This was discussed a bit in this attempt and another example of such a version bump is taskwarrior3, and the confusion rising in this attempt to bump the version of the main attribute.
Hmm, we can’t seem to get away from the fact that the package system doesn’t have actual version management. But that’s not necessarily a a negative.
Bit of a story time: I’m thinking of back when I was working with PHP, Composer ended up being one of (if not the) best examples of version management done well (in software overall, not just in PHP). PHP packages ecosystem ended up heavily relying on semver, and it’s one of the only places i’ve seen semver actually work. Every couple of weeks I used to go over our projects and bump all the minor and patch versions, and do a release, mostly without issues (along with checking the changelog via github’s diff in the url feature). The experience was smooth because the tooling and the community happen to work well together, and everyone was mostly following and benefiting from semver principles.
Just to close this off, I’ve never seen semver work well in any other other situation: either dependencies bump major frivolously, or they release breaking changes as minors, and those aren’t trivial mistakes like they seem at first.
So going back to nixpkgs and its approach of working towards one (or just over) working release version is great use of resource, especially because you can always go back in time and use an older packaged release just as it was then, it’s a neat way to use its strength without creating more maintenance work.
I think what’s currently missing is tooling to interrogate package versions (there’s a nice website I came across that does it Nix Package Versions) and help with diffing changes, and simple way of pinning packages. I think solution here is more information and lean further into “every package is highest release version” sort of thing. Perhaps even make use of diverging branches in github to deal with multiple supported version of the same package /shrug.
Sorry for the wall of text, hopefully that was useful
Another way is to maintain package identity by never changing the names, such that pkg2 will keep its name forever. This way, a bit ugly as it seems, avoids difficult questions like whether a few months is sufficient. I only update the packages in my system when a new version of NixOS is released, and I even had an experience where my system did not receive any updates for several years.
The unfortunate confusion raised in my PR was because I considered all-packages.nix as the single source of truth of all packages and only looked up the package in the source code. I overlooked the (other breaking) change that newer by-name packages are normally absent from this file. Now that I have refreshed my knowledge, this kind of confusion will never happen.
Keeping pkg1 along side pkg2 forever doesn’t solve the main issue I tried to present here - the fact that no body will be aware of the new version unless they’d check with upstream by themselves. In general, the fact that Nixpkgs fails to distribute the fact that there is a new version of a program (whether the new version is breaking or not) is a failure of the mission of the distribution in my opinion.
if pkg was a throw then users will know there is a newer version.
you only need to keep pkg1 and pkg2 for while pkg2 is considered “breaking”. Once it’s resolved ( imaginary perfect world ) pkg becomes alias to pkg2 and pkg1 becomes a throw to use pkg
It would be interesting if we can come up with versioning scheme that solves it, but current approach, while imperfect is simple. ( yes I also made PR trying to upgrade transmission 3 to 4 )
This is already what we strive for. Usually, if there are multiple versions, there is a reason for that. Most commonly it will be one of the following:
it is non-leaf package, it breaks BC, dependents do not support the new version, and removing them is undesirable (or, in special cases like LLVM or PHP, we care about breaking external consumers)
it has known issues or not considered stable yet, but we still want to have the new version
migration requires previous versions (e.g. NextCloud or bootstrapping compilers)
And going back works for one-off usages but, if you need the older software long term, the old Nixpkgs revision will quickly accumulate known security vulnerabilities, so you will need to resolve to out-of-tree overlays (example).
I still think this is what release notes are for. After all, we cannot distinguish whether user knows about the new version, and renaming the attribute just to let them know adds needless churn.
We just need to figure out how to make release notes work on unstable.
I would say this protocol makes sense for when the old attribute becomes deprecated (see also doc: add instructions to rename a package · Issue #142873 · NixOS/nixpkgs · GitHub). I think most people will agree warnings are the perfect method to let users know about deprecation in advance, without immediately breaking their configs.
Unfortunately, warnings also have side effects – they spam nix-env users, and I vaguely recall there was some issue with Hydra. Though, since warnings have been used in aliases.nix for a while now, perhaps Hydra is no longer a problem (it definitely should not be if we merge release: disallow aliases by jtojnar · Pull Request #316680 · NixOS/nixpkgs · GitHub), and nix-env spam might be considered an acceptable trade-off.
A silly question: Is throw meant to enforce an update? I ask because it seems throw makes installing the old version impossible due to the errors.
I am not experienced enough to make general statements about updates. However, in this case, I concluded I would not choose the newer version of taskwarrior after trying it out for some time because it removed the ability to save plain text files, making it hard to have version control over the data.
The assumption that newer versions are superior does not always hold. At least in the short run, perhaps it is better to have notices than to throw hard errors about updates.
Not necessarily enforce an update, but it is meant to enforce the user to rename the attribute in their configuration and thus to knowingly choose either the old or the new version.
I agree that it depends on upstream, and indeed using lib.warn seems to be more agreed upon as a general recommendation - that’s what I implemented in #258058.