Often (maybe even almost always) when I want to use geckodriver from a channel (in my case usually the current NixOS release, but I think nixos-unstable and NixPkgs master do have the same issue), that channel provides it in a version incompatible to the firefox version of the same channel. Each time (IIRC), I’ve filed a PR to update geckodriver to a compatible version.
Today I wanted to use chromedriver and noticed that it, too, was in a version incompatible to that of chromium in nixos-19.09, nixos-unstable and NixPkgs master. So I’ve filed NixOS/nixpkgs#85302.
Is there a good way to keep WebDrivers better in sync with the corresponding browsers in NixPkgs? I think we need a way to notice when a WebDriver has to be updated due to a browser version bump. (And by “notice” I mean before an end-user tries to use it.)
I can imagine some different approaches to achieve this:
Functional tests in browser derivations
Have the testPhase of browser packages depend on the corresponding WebDriver and actually use it in a test.
Pro
This also tests whether the browser actually works
Con
Requires all browsers that are to be controlled by the same WebDriver to have versions compatible with the same WebDriver version.
What about browsers that don’t have a headless mode? Requiring a display and/or opening a window during a test probably isn’t a good idea at all.
Sometimes browser version bumps probably have to happen quickly for security reasons. Having to also update the WebDriver right away as a hard requirement might be detrimental and will probably lead to that test just being commented out and then forgotten.
Functional tests NixOS tests
For each browser controllable by a WebDriver, have a NixOS test that tests that browser with that WebDriver.
Pro
This also tests whether the browser actually works
NixOS tests run in VMs, so they can probably use the VM’ss display for browsers without headless mode. (Though I didn’t get that to work when I tried with Firefox and geckodriver some time ago.)
Con
Are these tests ever ran outside of cutting a NixOS release?
Requires all browsers that are to be controlled by the same WebDriver to have versions compatible with the same WebDriver version.
Functional tests in WebDriver derivations
Have the testPhase of WebDriver packages depend on the corresponding browser and actually test that browser it in a test using the WebDriver.
Pro
Would hard-block browser version bumps, but breaking tests could be noticed with nixpkgs-review
Con
For WebDrivers that can control browsers from several packages, which one should they test?
What if that browser doesn’t have a headless mode? Requiring a display and/or opening a window during a test probably isn’t a good idea at all.
Each version of ChromeDriver supports Chrome with matching major, minor, and build version numbers. For example, ChromeDriver 73.0.3683.20 supports all Chrome versions that start with 73.0.3683.
This could be checked for in an automated test. (Dunno whether in the browser package, the WebDriver package or as a NixOS test and what the respective "pro"s and "con"s would be.)
Pro
computationally cheap
Con
What about Firefox / geckodriver? Would probably need a lookup table there.
Package WebDriver with the browser
Put into the package of each browser that can be controlled by a WebDriver the corresponding WebDriver into the same derivation. (Probably as a separate output, so that it doesn’t get installed by default.)
Pro
when different versions of the same WebDriver are needed by different browsers (or by different versions of the same browser provided within the same channel), this represents this best
Con
???
Comment in browser package source
Have a comment in the browser package NixPkgs source near to where the version(s) is/are defined to remind contributors changing those versions to also check the corresponding WebDriver.
Pro
Can easily be ignored, if appropriate.
Con
Can easily be ignored, even when not appropriate.
Is there any better way? Which of these ways would be most feasible, which one will be most maintainable?
I think adding the test to the webdriver makes the most sense. Hopefully we can rely on people using nixpkgs-review.
Regarding the issue of multiple browser versions: If we have multiple browser versions, the corrensponding webdriver (and other browser-related package) should probably be a subpackage of the browser.
Like we can use python2.pkgs.numpy and python3.pkgs.numpy we should have firefox42.pkgs.webdriver and firefox43.pkgs.webdriver. Note that I don’t know anything about how firefox is managed in nixpkgs, so maybe that doesn’t make any sense.