Overriding package not working as expected

I can tell this is trying to work, just not sure what is really wrong. I’m trying to override Handbrake to have Intel QSV support, so I added the configure flag, then it was saying it couldn’t find headers so I added intel-media-sdk. Right now the error is ‘build input intel-media-sdk does not exist’. Code:
nixpkgs = {
config = [
allowUnfree = true;
packageOverrides = pkgs: {
handbrake = pkgs.handbrake.overrideAttrs (old: {
configureFlags = old.configureFlags ++ [ “–enable-qsv” ];
buildInputs = old.buildInputs ++ [ “intel-media-sdk” ];
});
};
};
};

nixpkgs = {
  config = {
    allowUnfree = true;
    packageOverrides = pkgs: {
      handbrake = pkgs.handbrake.overrideAttrs (old: {
        configureFlags = old.configureFlags ++ [ “–enable-qsv” ];
        buildInputs = old.buildInputs ++ [ pkgs.intel-media-sdk ];
      });
    };
  };
};

Did you test build this? I see the changes you made, but wasn’t able to get this working. Also found some other small errors in my configure flag. Right now this sits as below but complains of the same dependency issues as it did before.

       > ../libhb/handbrake/ports.h:30:10: fatal error: vpl/mfxstructures.h: No such file or directory
       >    30 | #include "vpl/mfxstructures.h"
       >       |          ^~~~~~~~~~~~~~~~~~~~~


    config = {
      allowUnfree = true;
      packageOverrides = pkgs: {
        handbrake = pkgs.handbrake.overrideAttrs (old: {
          configureFlags = old.configureFlags ++ [ "--enable-qsv" ];
          buildInputs = old.buildInputs ++ [ pkgs.intel-media-sdk ];
        });
      };
    };

No, that was mostly to point out the syntax error you made to get you past that blocking issue.

Looks like you have some digging to do to find out what package provides that header. A quick search suggests you might need libvpl, which would match the folder name (note stable does not have this package yet).

Ok that makes sense. I was thinking something along those lines but couldn’t find libvpl packaged, didn’t think to look at unstable. I’m using misterio77’s templates. He has an overlay for unstable packages, but not working for me. I replaced pkgs.intel-media-sdk with pkgs.unstable.libvpl but get an error “error: attribute ‘unstable’ missing”

That’ll be because you’re not adding unstable to your packages yet. How to add unstable is a different question. Overlays are one way, but they’re a bit overkill for this (and I consider them a bit of an anti-pattern these days, if you have an overlay-shaped hammer everything seems to be a nail to some people).

If you’re using channels (and not flakes), a simple solution is something like this:

{ pkgs, ... }: let
  nixos-unstable = import <nixos-unstable> { };
in {
  nixpkgs = {
    config = {
      allowUnfree = true;
      packageOverrides = pkgs: {
        handbrake = pkgs.handbrake.overrideAttrs (old: {
          configureFlags = old.configureFlags ++ [ “–enable-qsv” ];
          buildInputs = old.buildInputs ++ [
            nixos-unstable.libvpl
          ];
        });
      };
    };
  };
}

You’d also need to add the unstable channel to your system:

sudo nix-channel --add https://nixos.org/channels/nixos-unstable nixos-unstable

caveat emptor: I haven’t used channels in a long time, this is coming from memory

For flakes you’d add unstable as an input and follow one of the strategies in this blog post (my favorite is just adding the inputs to specialArgs): Getting inputs to modules in a nix-flake | NobbZ' Blog

Either way, if you don’t get a working result immediately following this, I’d encourage you to learn how channels work before going too far down this rabbit hole without a guide. Or flake inputs, if you use flakes.

YMMV in the end, customizing packages is hard if you’re not the package author. Consider upstreaming whatever you end up with as an overrideable option argument to the package so you can just set pkgs.handbrake.override { qsvSupport = true; } (or nixos-unstable.handbrake.override { qsvSupport = true; } if you stick to stable), then you don’t need to maintain it yourself in the future.