Running playwright tests

Help
21

I’m glad my comment was helpful!

If you’re having trouble finding the nixpkgs’s commit in which the desired playwright version you might find it useful nixhub

How I use it:

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    nixpkgs-pw-v1_40.url = "github:nixos/nixpkgs/0a254180b4cad6be45aa46dce896bdb8db5d2930";
  };

  outputs = { self, nixpkgs, nixpkgs-pw-v1_40 }: let
    system = "x86_64-linux";
    pkgs = import nixpkgs { inherit system; };
    pkgs-pw-v1_40 = import nixpkgs-pw-v1_40 { inherit system; };
  in {
      devShells.${system}.default = pkgs.mkShell {
        packages = [
          pkgs.nodejs_20
          pkgs-pw-v1_40.playwright
        ];

        PLAYWRIGHT_NODEJS_PATH = "${pkgs.nodejs_20}/bin/node";
        PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS = true;

        PLAYWRIGHT_BROWSERS_PATH="${pkgs-pw-v1_40.playwright-driver.browsers}";
    };
  };
}

For example: if I need to use pw 1.38 instead 1.40, I’ll just use commit a71323f68d4377d12c04a5410e214495ec598d4c (found it on nixhub).

What would need to change to make this work with direnv?

My .envrc in project folder:


if ! has nix_direnv_version || ! nix_direnv_version 3.0.4; then
    source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/3.0.4/direnvrc" "sha256-DzlYZ33mWF/Gs8DDeyjr8mnVmQGx7ASYqA5WlxwvBG4="
fi
use flake /src/flakes/flake-for-this-project

I store flakes separately from projects (/src/flakes/*), but of course it’s easier to keep them in a project.

22

Same issue for me. I’ve been really careful with the versions:
pnpm list shows playwright 1.40.0 and echo ${pkgs.playwright-driver.browsers.version} in the shellHook shows 1.40.0. However pnpx playwright codegen results in

Error: t.parse: Executable doesn't exist at
/nix/store/xc7q18y0rw1yk40wrvqxzy87fa3vzccr-playwright-browsers-chromium/chromium-1105/chrome-linux/chrome

But

/nix/store/xc7q18y0rw1yk40wrvqxzy87fa3vzccr-playwright-browsers-chromium/chromium-1091/chrome-linux/chrome

does exist (chromium-1091, not 1105), which is kind of weird too…

Any help would be appreciated.

1 Like
23

Strange, if the versions of playwright are really the same, the same browser would have been used. :thinking:

24

Do these solutions work with Firefox as well? Or just with Chromium?

25

I was only able to do it with Chromium, Firefox failed.

26

Okay, it seems things are working now. As it turns out, it looks like playwright-driver.browsers only provides chromium, not firefox or webkit. Thus, I was getting the following stack trace, and focused on the “executable doesn’t exist” errors and missed the “1 passed” at the end.

It still sucks that I can’t test my software on firefox or webkit, but now I guess it’s the Playwright team’s responsibility to make their project work on Linux?

Anyway, thank you for your help!

❯ pnpm exec playwright test

Running 3 tests using 2 workers
  1) [firefox] › example.spec.ts:3:5 › test ────────────────────────────────────────────────────────

    Error: browserType.launch: Executable doesn't exist at /nix/store/jws56gk36j4akxcv4xisad8ak2wib5sd-playwright-browsers-chromium/firefox-1429/firefox/firefox
    ╔═════════════════════════════════════════════════════════════════════════╗
    ║ Looks like Playwright Test or Playwright was just installed or updated. ║
    ║ Please run the following command to download new browsers:              ║
    ║                                                                         ║
    ║     pnpm exec playwright install                                        ║
    ║                                                                         ║
    ║ <3 Playwright Team                                                      ║
    ╚═════════════════════════════════════════════════════════════════════════╝




  2) [webkit] › example.spec.ts:3:5 › test ─────────────────────────────────────────────────────────

    Error: browserType.launch: Executable doesn't exist at /nix/store/jws56gk36j4akxcv4xisad8ak2wib5sd-playwright-browsers-chromium/webkit-1944/pw_run.sh
    ╔═════════════════════════════════════════════════════════════════════════╗
    ║ Looks like Playwright Test or Playwright was just installed or updated. ║
    ║ Please run the following command to download new browsers:              ║
    ║                                                                         ║
    ║     pnpm exec playwright install                                        ║
    ║                                                                         ║
    ║ <3 Playwright Team                                                      ║
    ╚═════════════════════════════════════════════════════════════════════════╝




  2 failed
    [firefox] › example.spec.ts:3:5 › test ─────────────────────────────────────────────────────────
    [webkit] › example.spec.ts:3:5 › test ──────────────────────────────────────────────────────────
  1 passed (4.6s)

  Serving HTML report at http://localhost:9323. Press Ctrl+C to quit.
27

It DOES work on Linux on distros like Debian / Ubuntu… :wink: Playwright is downloading the browser binaries via NPM and running them directly. But NixOS of course has troubles with binaries that depend on all kind of paths that we don’t have…

1 Like
28

could you try and comment on Playwright: browser improvements, update by phaer · Pull Request #298944 · NixOS/nixpkgs · GitHub that brings firefox support ?

2 Likes
29

I managed to get this working through some of the suggestions, and one additional tweak. I do not have to rely on the versions being aligned as I explicitly set the executablePath for the launch options.

flake.nix

         packages = with pkgs; [
            nodejs_21

            # playwright
            playwright-driver.browsers
          ];

          # playwright
          PLAYWRIGHT_NODEJS_PATH = "${pkgs.nodejs_21}/bin/node";
          PLAYWRIGHT_LAUNCH_OPTIONS_EXECUTABLE_PATH = "${pkgs.playwright-driver.browsers}/chromium-1091/chrome-linux/chrome";
          PLAYWRIGHT_BROWSERS_PATH = "${pkgs.playwright-driver.browsers}";

package.json

    "@playwright/experimental-ct-svelte": "^1.43.0",
    "@playwright/test": "^1.43.0",

playwright.config.ts

projects: [
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        launchOptions: {
          executablePath: process.env.PLAYWRIGHT_LAUNCH_OPTIONS_EXECUTABLE_PATH as string,
        }
      },
    },
]
3 Likes
30

But doesn’t this mean that you potentially use a different browser version than the one coming with your @playwright/test version which may lead to problems.

When testing with playwright locally, I usually set the playwright version of the project to the one coming with Nixpkgs so that the browser version align.

31

Obviously the preference is to run the same versions. But playwright-driver is currently only set to 1.40 and my requirements were to use playwright-ct 1.43 due to some changes that were made. The only way to get around it was to force a browser version.

The issues i’ve run into are that ui and debug versions don’t seem to work.

32

If you want you can grab that 1091 programmatically:

shellHook = ''
  playwright_chromium_revision="$(${jq}/bin/jq --raw-output '.browsers[] | select(.name == "chromium").revision' ${playwright-driver}/package/browsers.json)"

  export PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH="${playwright-driver.browsers}/chromium-$playwright_chromium_revision/chrome-linux/chrome";
'';
2 Likes
33

Pure nix solution :):

{
  devenv,
  inputs,
  pkgs,
  ...
}:
{
  my-shell = devenv.lib.mkShell {
    inherit inputs pkgs;
    modules =
      let
        playwright-driver = pkgs.playwright-driver;
        playwright-driver-browsers = pkgs.playwright-driver.browsers;

        playright-file = builtins.readFile "${playwright-driver}/package/browsers.json";
        playright-json = builtins.fromJSON playright-file;
        playwright-chromium-entry = builtins.elemAt (builtins.filter (
          browser: browser.name == "chromium"
        ) playright-json.browsers) 0;
        playwright-chromium-revision = playwright-chromium-entry.revision;
      in
      [
        {
          # TODO: add MacOS support to omit this
          env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH = "${playwright-driver-browsers}/chromium-${playwright-chromium-revision}/chrome-linux/chrome";
          # This is used by npx playwright --{ui,debug,...}
          env.PLAYWRIGHT_BROWSERS_PATH = "${playwright-driver-browsers}";
          languages = {
            javascript = {
              enable = true;
              npm.enable = true;
            };
          };
        }
      ];
  };
}

I use devenv, but this should work with anything. And in your playwright.config.ts, have something like this:

import { defineConfig, devices } from '@playwright/test';
import dotenv from 'dotenv';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

dotenv.config({ path: path.resolve(__dirname, '.env') });

/**
 * See https://playwright.dev/docs/test-configuration.
 */
export default defineConfig({
  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        launchOptions: {
          ...(process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH
            ? {
                executablePath: process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH,
              }
            : {}),
        },
      },
    },
  ],
 }
)

This makes it so only if the executable path is set then it will use it (why does it not do this by default? idk…) :face_with_spiral_eyes:

You can also use this method to launch playwright directly (index.js):

import { chromium } from 'playwright-core';
import dotenv from 'dotenv';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

dotenv.config({ path: path.resolve(__dirname, '.env') });

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

const main = async () => {
  const patchOptions = {
    ...(process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH
      ? { executablePath: process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH }
      : {}),
  };
  const browser = await chromium.launch({
    ...patchOptions,
    headless: false,
  });
  const context = await browser.newContext();
  const page = await context.newPage();
  await page.goto('https://playwright.dev');
  await sleep(5000);
  await browser.close();
};

await main();

Although, this only lets you use chrome. This is fine for me, for now. I think I will try and resolve the lib error and prefer the browsers playwright installs. I don’t want to have to override the browser… :frowning:

2 Likes
34

Very helpful thanks. Added a bit of logic to support mac as well

  inherit (pkgs.stdenv.hostPlatform) isLinux;

  ///...same as above...///

  browserBin =
    if isLinux then "${playwright-driver.browsers}/chromium-${playwright-chromium-revision}/chrome-linux/chrome"
    else "${playwright-driver.browsers}/chromium-${playwright-chromium-revision}/chrome-mac/Chromium.app/Contents/MacOS/Chromium";


PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH = "${browserBin}";
1 Like
35 
let pkgs = import <nixpkgs> {};
in 

pkgs.buildFHSEnv {

  name = "playwright";

  targetPkgs = pkgs: with pkgs; [

    # playwright (chrome headless = false works)
    openssl 
    systemd 
    glibc glibc.dev

    glib
    cups.lib cups
    nss
    nssTools
    alsa-lib
    dbus
    at-spi2-core
    libdrm
    expat
    xorg.libX11
    xorg.libXcomposite
    xorg.libXdamage
    xorg.libXext
    xorg.libXfixes
    xorg.libXrandr
    xorg.libxcb
    mesa
    libxkbcommon
    pango
    cairo
    nspr
  ];

  profile = ''
    export LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib:/lib
    export FONTCONFIG_FILE=/etc/fonts/fonts.conf
  '';

  runScript = pkgs.writeShellScript "wrapper" ''
    exec ${pkgs.zsh}/bin/zsh
  '';

  # As intended by this bubble wrap, share as much namespaces as possible with user.
  unshareUser   = false;
  unshareIpc    = false;
  unsharePid    = false;
  unshareNet    = false;
  unshareUts    = false;
  unshareCgroup = false;
  # Since "insync start" command starts a daemon, this daemon should die with it.
  dieWithParent = true;
}

nix-build (nixpkgs-23.11) then chroot. Then simple test

import { chromium, devices } from 'playwright';
import assert from 'node:assert';

(async () => {
    // Setup
    const browser = await chromium.launch({ headless: false });
    const context = await browser.newContext(devices['iPhone 11']);
    const page = await context.newPage();

    // The actual interesting bit
    await context.route('**.jpg', route => route.abort());
    console.log(1);
    await page.goto('https://example.com/');
    console.log(2);

    const pt = await page.title()
    console.log("page title");
    console.log(JSON.stringify(pt));

    assert(pt === 'Example Domain'); // 👎 not a Web First assertion

    const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
    await sleep(5000)

    // Teardown
    await context.close();
    await browser.close();
})();

with chrome worked with the npx playwright install command meaning you can use the version the project was tested with.

36

I’ve found a solution with Nix flakes. This is for running Playwright tests in a .NET project, but it can perhaps be slightly adapted for Java/Python/JS.

Here’s the flake.nix I’ve used. Execute nix develop to launch the development environment.

{
  description = ".NET development environment based on the Filesystem Hierarchy Standard (FHS)";
  # https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
  };

  outputs = { nixpkgs, ... } @ inputs:
  let
    system = "x86_64-linux";
    pkgs = nixpkgs.legacyPackages.${system};

    # .NET SDKs
    dotnet_sdk = (with pkgs.dotnetCorePackages; combinePackages [
      # Install all the .NET SDK versions needed here...
      # sdk_8_0
      # sdk_7_0
      sdk_6_0
    ]);
  in
  {
    # Create development shell based on the Filesystem Hierarchy Standard (FHS) with a set of
    # standard packages based on the list maintained by the appimagetools package
    #
    # buildFHSEnv -> https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environments
    #
    # The packages included in appimagetools.defaultFhsEnvArgs are:
    # https://github.com/NixOS/nixpkgs/blob/fd6a510ec7e84ccd7f38c2ad9a55a18bf076f738/pkgs/build-support/appimage/default.nix#L72-L208
    devShells.${system}.default = (pkgs.buildFHSEnv (pkgs.appimageTools.defaultFhsEnvArgs // {
          name = "dotnet-development-environment";
          # Packages installed in the development shell
          targetPkgs = pkgs: with pkgs; [
            # .NET SDK
            dotnet_sdk
            # Run PowerShell scripts, which are sometimes included in NuGet packages like Playwright
            powershell
            # Timezones
            tzdata
            # Locales
            glibcLocales
          ];
          # Commands to be executed in the development shell
          profile = ''
            # Ensure that dotnet tools can find the .NET location
            export DOTNET_ROOT="${dotnet_sdk}";

            # Set LANG for locales, otherwise it is unset when running "nix-shell --pure"
            export LANG="C.UTF-8"

            # Remove duplicate commands from Bash shell command history
            export HISTCONTROL=ignoreboth:erasedups

            # Do not pollute $HOME with config files (both paths are ignored in .gitignore)
            export DOTNET_CLI_HOME="$PWD/.net_cli_home";
            export NUGET_PACKAGES="$PWD/.nuget_packages";

            # Put dotnet tools in $PATH to be able to use them
            export PATH="$DOTNET_CLI_HOME/.dotnet/tools:$PATH"
          '';
        })).env;
  };
}

As you can see, no environment variables are defined for Playwright nor do you need to install playwright-driver to use patched browsers. With a devShell relying on the Filesystem Hierarchy Standard and standard packages provided by appimageTools.defaultFhsEnvArgs, it’s possible to follow the installation instructions from Playwright. You can install the browsers provided by Playwright and they work. On top of this, you don’t have to rely on the playwright-driver package which is sometimes a few updates behind the latest Playwright version. You also get all browsers, not only Chromium which is the single browser supported in playwright-driver at the time of writing.

1 Like
37

In case anyone finds this. I was able to install chromedriver in home-manager, and then I was able to run standard npx playwright test on Ubuntu 22.04 without any of the errors related to " Host system is missing dependencies to run browsers".

38

If you are on Ubuntu, I guess Playwright downloads, installs and uses the usual binaries from somewhere. Is that so? The problem arises when you are on NixOS where you can not run binaries which expect an FHS environment. That’s why we have all the trouble.