Gnome extensions, updates, and testing

uep · November 15, 2021, 10:07pm

A little while ago, nixos-unstable updated to gnome 41, and in the process a number of the extensions I like broke:

This isn’t entirely unexpected; each extension has to list the versions it’s compatible with in a metadata.json file, and some authors and maintainers are faster than others. Possibly exacerbated because I don’t think that many “mainline” distributions are using 41 (or even 40) yet.

Even though some of these have text descriptions with comments like “for gnome 40 and above”, the actual metadata doesn’t cover “above”. For several of these, I either know or suspect that the extension will work fine with just the metadata update. I’d like to help the authors by testing this and sending a PR.

This turns out to be… kinda complex on NixOS. Nearly all of the extensions are built programmatically from yet another json manifest, so there’s no simple package I can edit to patch the metadata.json file. Any scheme I can come up with (other than force editing the nix store) is just loads more effort than seems sensible for what it ultimately a one-line change in the upstream source.

Is there a way to inject some additional synthetic metadata into the programmatic builds, so that I can:

test the change locally, quickly, and simply
(optionally) PR the change to nixpkgs until the upstream makes a new release
PR a change, with confirmation of testing, to the upstream extension

(Prompted to post in part by: Cannot enable dash-to-dock gnome extension)

uep · November 15, 2021, 10:34pm

There’s already a version map in the big ball of json that the automatic build mechanism uses, which is presumably derived from the extension data (from several releases).


  "description": "...", 
  "name": "...", 
  "shell-version": [
    "3.34",
    "3.36",
    "3.38",
    "40",
    "41"
  ],
  "url": "...", 
  "uuid": "...", 
  "version": 21
}

Given the build knows which version of gnome is in use, and the nix pkg will be built for one specific configuration, with an override the existing shell-version list in the output metadata.json could be entirely overridden with just the target version?

jtojnar · November 15, 2021, 11:16pm

First, it is possible to disable the version validation using gsettings set org.gnome.shell disable-extension-version-validation true. Just note that using incompatible extensions might cause shell instability or other issues.

Then you can run journalctl -xeaf and check logs for errors occurring on enabling the extension and throughout its use. You should note the errors and warnings you discover, as well as the interactions leading to them when reporting upstream.

If I want to tweak the extension source to help porting it to a new shell version, I typically clone its repository locally and then apply a patch like this to my local Nixpkgs checkout:

--- a/pkgs/desktops/gnome/extensions/sound-output-device-chooser/default.nix
+++ b/pkgs/desktops/gnome/extensions/sound-output-device-chooser/default.nix
@@ -9,12 +9,7 @@ stdenv.mkDerivation rec {
   pname = "gnome-shell-extension-sound-output-device-chooser";
   version = "39";
 
-  src = fetchFromGitHub {
-    owner = "kgshank";
-    repo = "gse-sound-output-device-chooser";
-    rev = version;
-    sha256 = "sha256-RFdBdpKsz2MjdzxWX4UFwah+e68dqrkvm7ql0RAZZwg=";
-  };
+  src = /home/jtojnar/Projects/gse-sound-output-device-chooser;
 
   patches = [
     # Fix paths to libpulse and python

Then I install the extension using nix-env -f . gnomeExtensions.sound-output-device-chooser and restart the shell by pressing Alt-F2 and entering r into the entry (only works in Xorg session, unfortunately).

dbus-run-session -- gnome-shell --nested might also work for some lighter testing.

Unfortunately, this is much less convenient for automatically packaged extensions, as you basically need to re-create the manual packaging. For example, for transparent-window-moving:

--- a/pkgs/desktops/gnome/extensions/extensionOverrides.nix
+++ b/pkgs/desktops/gnome/extensions/extensionOverrides.nix
@@ -1,6 +1,7 @@
 { lib
 , ddcutil
 , gjs
+, glib
 , xprop
 }:
 # A set of overrides for automatically packaged extensions that require some small fixes.
@@ -11,6 +12,20 @@
 # the upstream repository's sources.
 super: super // {
 
+  "transparent-window-moving@noobsai.github.com" = super."transparent-window-moving@noobsai.github.com".overrideAttrs (old: {
+    src = /home/jtojnar/Projects/transparent-window-moving;
+    nativeBuildInputs = [
+      glib # for glib-compile-schemas
+    ];
+    makeFlags = [
+      "EXTENSIONDIR=${placeholder "out"}/share/gnome-shell/extensions"
+    ];
+    # Override the default attributes set by buildShellExtension.
+    version = "9";
+    dontBuild = false;
+    installPhase = null;
+  });
+
   "dash-to-dock@micxgx.gmail.com" = super."dash-to-dock@micxgx.gmail.com".overrideAttrs (old: {
     meta.maintainers = with lib.maintainers; [ eperuffo jtojnar rhoriguchi ];
   });

Alternately, you can just get the source from the extension portal using cp -r $(nix-build -A gnomeExtensions.transparent-window-moving.src) transparent-window-moving && chmod -R +w transparent-window-moving and just point src attribute in extensionOverrides.nix to that but then you might have harder time porting the changes back to the original source code.

uep · November 15, 2021, 11:56pm

Well, that’s good to know. And at least I can probably disable just some of the extensions that have actual problems. Edit: so far it seems like none of them, this time.

The rest of your reply is a really nice worked-example of … exactly the problem I was getting at. Thankyou — this was the kind of approach I was visualising, but hadn’t put into practice — but I think my main point stands.

I’ve just looked more closely at .../extensions/buildGnomeExtension.nix, and learned what the encoded stuff in the big ball of json contains:

extraPostFetch = ''
  echo "${metadata}" | base64 --decode > $out/metadata.json
'';

So it’s already (very lightly) manipulating the metadata.json file.

Would it be feasible to have buildGnomeExtension look at a key, that could be set in extensionOverrides.nix, that edits a specific value / current target gnome version, into the shell-version field of that output file?

Failing that / in the meantime, I guess I can replace the base64 content of that item in extensions.json in a local nixpkgs repo. I suspect it’s actually not part of the source hash after all.

jtojnar · November 16, 2021, 12:27am

The fetchzip function produces a fixed-output derivation and the extraPostFetch is definitely counted in the source hash. Its whole point is to make the extension sources deterministic – the extension portal occasionally changes the metadata in existing zip files, which would break the hash. For that reason, changing the metadata in extensions.json without also updating sha256 will not work – Nix will use the old source since that hash already exists in the store.

It would be feasible but would need to be done in buildShellExtension, not the fetchzip, to avoid having to store per-version hashes.

What do you think @piegamesde?

piegames · November 16, 2021, 11:18am

Some random thoughts:

First of all I’d like to highlight how much of all of this is just hacking around the limitations of extensions.gnome.org. Especially the metadata pinning in the extensions.json is something that I really dislike.
Similarly, all our extensions are currently not built from source. Since JavaScript is an interpreted language, this doesn’t make a huge difference conceptually. However it means that our “sources” don’t match the upstream’s sources, they instead match upstream releases. This makes patching automatically packaged extensions so annoying: You either have to add a manual package (which is just boilerplate on most extensions, except those that use TypeScript or so), or you have to adapt your patch to the slightly different source. Upstream has plans for having a more unified build pipeline across extensions, but it’s probably going to take a long time and until then there’s not really much we can do.
Personally, my preferred solution would be to simply have users disable-extension-version-validation, because it introduces the least maintenance overhead. It also does not put us at risk of implicitly breaking things (the user would be more clear about the fact that it’s their fault ^^). But I understand your motivation for a nixpkgs solution.

Now onto the question:

There are many possible locations for patching the information in the pipeline. We first of all need to tell our packaging that the extension actually supports GNOME 41. I’d suggest an attribute override to extensions.json. Then, we also need to tell GNOME the same in the metadata.json. Although that file is technically contained in extensions.json, overriding it there would not be a good idea: it is hex encoded in a string and modifying it would give us hash mismatch headaches again. I think a better way would be to inject a patch or a fixup hook to that file in an override of the extension.

Then, there’s the problem of maintainability. We would want to be informed somehow that the extension now supports that GNOME version upstream, and that we thus have to remove our patch again. I don’t have any ideas off the top of my hat for this.

You see, it’s technically possible, but it won’t be easy for sure. I wouldn’t be against it provided that maintainability is preserved, but I would prefer some solution that cooperates with GNOME to improve the general situation (e.g. to make sure extensions are already up to date when GNOME updates).

uep · November 16, 2021, 8:54pm

I wasn’t aware of this option previously, and (at least in this iteration) it’s worked superbly – so frankly it’s my preferred solution now too. As you note, the other motivation is still there, but now far more … esoteric and academic, as much or more for my own learning about nix than for any remaining practical use.

I’m still trying to convince myself how and where the base64 decode into the output metadata has to be the same as the input from the zip that went into the hash, and then into the extensions.json in the python script, but again that’s now for my own learning and curiosity more than anything.

This workaround is also useful enough that the extra overhead of the more manual overrides (like making a full package to replace the automated one, just to be able to apply a patch) is likely to be rare, or needed anyway because there are other patches needed to send upstream to fix compatibility.

Very much so.

piegames · November 16, 2021, 11:49pm

Okay, let me explain: Everything starts in the update-extensions.py. That script scrapes the extensions.gnome.org database through their (more or less unofficial) search API. This gives us the preliminary version information. From there, we download each extension in all version that we are interested in, and store the URL and the hash alongside in our metadata.

Now it comes that somebody changes the description of their extension. This will be reflected in the metadata.json (it is partly automatically generated by the site), which will in turn cause some hash mismatch when building the extension the next time (figuring that one out involved a lot of headache and “WAT??”). Thus, our download script now also takes the metadata.json and stores it alongside our metadata as base64 string. (At first, I tried less invasive patching approaches with storing only certain keys, but to no veil.) Now, we only need to override that file when downloading it, but before the hash is calculated, so that the content and thus the hash match exactly what we expect. This involves a few more obscure fetchzip options that you already encountered.