Whilst I haven’t addressed the issue above, I thought I would document out how I am currently doing this before I move on to something else and whilst it is still fresh in my mind. I would really like to know; 1) if I have done something dumb, 2) if there is a better way or 3) if there is something I could do better (bash-scripting arggghh!). Always keen to learn. 
My objectives
- Don’t require the user to chose a specialisation at boot.
- Use a Thunderbolt eGPU with a NVidia graphics card if it is available
- Dynamically work out the PCIE address of the NVidia card so I can plug it in to any Thunderbolt port, even daisy-chained off a dock and it should work.
All these changes were done in configuration.nix.
Summary of the steps
- I start by creating a tmpfs (yes I know it’s not really a ramdisk)
- I then create a shell script and a systemd service to start it before the graphical target. This modifies the exported xorg.conf and updates the PCIe address.
- I then setup all my config as if my system were running all the video drivers.
- Lastly I tell X to export it config to /etc/X11 so I can modify it and tell X to use the modified one
Steps in Detail
Create /tmp/ramdisk u=root / g=root / 777. I do this using a system activationScript with some basic checks.
system = {
activationScripts = { # 11Feb24 : system.activationScripts run each boot and rebuild so must be idempotent
createFileSharePath = { # 11Feb24 : Create a script to create and setup folders
text = ''
MakeFolder() { # 11Feb24 : Define a subroutine to create folders and set permissions, owner and group
if [ ! -e $1 ]; then # 11Feb24 : Check if the entry doen't exist
mkdir $1 # 11Feb24 : Create the folder
fi
if [ -d $1 ]; then # 11Feb24 : Check if entry exists and is a directory
if [ $(stat -c %U "$1") != "$2" ]; then # 11Feb24 : Check if owner needs changing
chown $2 $1 # 11Feb24 : Change Owner
fi
if [ $(stat -c %G "$1") != "$3" ]; then # 11Feb24 : Check if group needs changing
chgrp $3 $1 # 11Feb24 : Change Group
fi
if [ $(stat -c %a "$1") != "$4" ]; then # 11Feb24 : Check if permissions needs changing
chmod $4 $1 # 11Feb24 : Change permissions
fi
fi
}
MakeFolder "/tmp/ramdisk" root root 777 # 26Jun24 : Create a folder to have a small ramdisk for temp files
'';
};
};
};
Create a temporary place to build the xorg.conf that is used to start x. It doesn’t need to survive between boots as it is built every time.
systemd = {
mounts = [
{
enable = true;
options = "size=64m";
type = "tmpfs";
wantedBy = [
"x-config-prepare.service"
];
what = "ramdisk";
where = "/tmp/ramdisk/";
}
];
};
Create the shell script to be started by a systemd service. The main things someone else needs to check here are;
- the string that matches the NVidia card:
hexNvidiaPcieAddress=$(${pkgs.pciutils}/bin/lspci | grep "VGA compatible controller: NVIDIA Corporation TU116 \[GeForce GTX 1650 SUPER] (rev a1)" | sed -r 's/^([0-9a-f:\.]+)\s.+$/\1/') - the decision to remove the frame buffer drivers at the end of the script. For me, this stops the laptop’s internal HDMI port from being availble. If you want the HDMI port just comment out that line
let
ShellScript-x-config-prepare-script = pkgs.writeShellScriptBin "x-config-prepare-script" ''
#!/usr/bin/env bash
#_Version_|_Date____|_Time__|_Who___________________|_Notes______________________
# 1.1.0 | 27Jun24 | 09:39 | fsbof | Initial Script
# 1.2.0 | 28Jun24 | 17:15 | fsbof | Revised to use /tmp/ramdisk
#
#_Todo___________________________________________________________________________
#
#
usage () {
echo
echo "Usage: $0 [-h|--help]"
echo
echo "This script will;"
echo "Copy the exported xorg.conf to the temporary location. This requires"
echo " services.xserver.exportConfiguration = lib"".mkForce true;"
echo " lib"".mkForce seems to be required on 23.11 but not 24.05."
echo "Get the current PCIE address of a specific eGPU/Thunderbolt connected"
echo " NVidia card, if it exists."
echo "Tidy and Strip out the ServerLayout, Screen and Devices sections of"
echo " the xorf.conf file."
echo "Appended the new ServerLayout, Screen and Device sections with the"
echo " correct PCIe address if required."
echo
echo "-h | --help : ignores any other parameters and returns this help file."
echo
echo "Exit Codes"
echo " 0 - Success"
echo
echo "Version 1.2.0 - 28Jun24 17:15"
}
for args in "$@"
do
if [ "$args" == "-h" ] || [ "$args" == "--help" ]
then
usage
exit
fi
done
# Copy the xorg.conf file to the ramdisk
cp /etc/X11/xorg.conf /tmp/ramdisk/orig-xorg.conf
# Check if the NVidia card is available by looking for its PCIe address
hexNvidiaPcieAddress=$(${pkgs.pciutils}/bin/lspci | grep "VGA compatible controller: NVIDIA Corporation TU116 \[GeForce GTX 1650 SUPER] (rev a1)" | sed -r 's/^([0-9a-f:\.]+)\s.+$/\1/')
# Tidy and Strip out the ServerLayout, Screen and Devices sections of xorf.conf
newXorgFile=/tmp/ramdisk/xorg.conf
# Initialize counters
inServerLayout=0
inScreen=0
inDevice=0
inSection=0
lastLineIsBlank=0
# Process the original Xorg file line by line
while IFS= read -r line; do
if [[ $inServerLayout == 1 ]]; then
if [[ $line =~ EndSection ]]; then
inServerLayout=0
fi
elif [[ $inScreen == 1 ]]; then
if [[ $line =~ EndSection ]]; then
inScreen=0
fi
elif [[ $inDevice == 1 ]]; then
if [[ $line =~ EndSection ]]; then
inDevice=0
fi
elif [[ $inSection == 1 ]]; then
if [[ $line =~ EndSection ]]; then
inSection=0
echo "$line" >> "$newXorgFile"
lastLineIsBlank=0
elif [[ ! $line =~ ^[[:space:]]*$ ]]; then
echo "$line" >> "$newXorgFile" # include the line if it is not blank and is in a section
lastLineIsBlank=0
fi
elif [[ $line =~ Section.*\"ServerLayout\" ]]; then
inServerLayout=1 # do not include these sections
elif [[ $line =~ Section.*\"Screen\" ]]; then
inScreen=1 # do not include these sections
elif [[ $line =~ Section.*\"Device\" ]]; then
inDevice=1 # do not include these sections
elif [[ $line =~ Section.*\" ]]; then
inSection=1 # In a Section
echo "$line" >> "$newXorgFile"
lastLineIsBlank=0
elif [[ $line =~ ^# ]]; then
: # don't include lines that are purely comments
elif [[ $line =~ ^[[:space:]]*$ ]]; then
if [[ $lastLineIsBlank == 0 ]]; then
echo "$line" >> "$newXorgFile" # include the line if it is blank and outside a section but stop next blank line
lastLineIsBlank=1
fi
else
echo "$line" >> "$newXorgFile"
fi
done < "/tmp/ramdisk/orig-xorg.conf"
# If there is a PCIe address for the nvidia, then lets use it
if [ "$hexNvidiaPcieAddress" == "" ]; then
# No NVidia card, so all that is needed is to append the ServerLayout, Screen and Device sections
echo "Section \"ServerLayout\"" >> "$newXorgFile"
echo " Identifier \"Layout[all]\"" >> "$newXorgFile"
echo " # Reference the Screen sections for each driver. This will" >> "$newXorgFile"
echo " # cause the X server to try each in turn." >> "$newXorgFile"
echo " Screen \"Screen-modesetting[0]\"" >> "$newXorgFile"
echo " Screen \"Screen-fbdev[1]\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "# For each supported driver, add a \"Device\" and \"Screen\"" >> "$newXorgFile"
echo "# section." >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Device\"" >> "$newXorgFile"
echo " Identifier \"Device-modesetting[0]\"" >> "$newXorgFile"
echo " Driver \"modesetting\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Screen\"" >> "$newXorgFile"
echo " Identifier \"Screen-modesetting[0]\"" >> "$newXorgFile"
echo " Device \"Device-modesetting[0]\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Device\"" >> "$newXorgFile"
echo " Identifier \"Device-fbdev[1]\"" >> "$newXorgFile"
echo " Driver \"fbdev\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Screen\"" >> "$newXorgFile"
echo " Identifier \"Screen-fbdev[1]\"" >> "$newXorgFile"
echo " Device \"Device-fbdev[1]\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
# Remove the nvidia drivers
${pkgs.gnused}/bin/sed -i '/-nvidia-x11-/d' $newXorgFile
else
# First convert the PCIE address to the format required by Xorg - Decimal,Colons,No Leading Zeros
hexNvidiaPcieAddress=$(${pkgs.gnused}/bin/sed -r 's/\./\:/g' <<< "$hexNvidiaPcieAddress") # convert the last dot to a colon
IFS=':' read -r -a hexNumbers <<< "$hexNvidiaPcieAddress" # read the string into an array seperated by colons
decimalNvidiaPcieAddress="" # create a new empty string variable
for hexNumber in "''${hexNumbers[@]}"; do # Loop through each hexadecimal number and convert it to decimal
hexNumber=$(echo "$hexNumber" | ${pkgs.gnused}/bin/sed 's/^0*//') # Strip leading zero
decimalNumber=$(printf "%d" "0x$hexNumber") # convert to decimal
decimalNvidiaPcieAddress="$decimalNvidiaPcieAddress""$decimalNumber"":" # add to new decimal string
done
decimalNvidiaPcieAddress=$(sed -r 's/\:$//' <<< "$decimalNvidiaPcieAddress") # remove the final colon
# Append the ServerLayout, Screen and Device sections with the right PCIE address
echo "Section \"ServerLayout\"" >> "$newXorgFile"
echo " Identifier \"Layout[all]\"" >> "$newXorgFile"
echo " Option \"AllowNVIDIAGPUScreens\"" >> "$newXorgFile"
echo " # Reference the Screen sections for each driver. This will" >> "$newXorgFile"
echo " # cause the X server to try each in turn." >> "$newXorgFile"
echo " Screen \"Screen-modesetting[2]\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "# For each supported driver, add a \"Device\" and \"Screen\"" >> "$newXorgFile"
echo "# section." >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Device\"" >> "$newXorgFile"
echo " Identifier \"Device-modesetting[2]\"" >> "$newXorgFile"
echo " Driver \"modesetting\"" >> "$newXorgFile"
echo " BusID \"PCI:0:2:0\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Screen\"" >> "$newXorgFile"
echo " Identifier \"Screen-modesetting[2]\"" >> "$newXorgFile"
echo " Device \"Device-modesetting[2]\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
echo "" >> "$newXorgFile"
echo "Section \"Device\"" >> "$newXorgFile"
echo " Identifier \"Device-nvidia[2]\"" >> "$newXorgFile"
echo " Driver \"nvidia\"" >> "$newXorgFile"
echo " BusID \"PCI:$decimalNvidiaPcieAddress\"" >> "$newXorgFile"
echo " Option \"AllowExternalGpus\"" >> "$newXorgFile"
echo "EndSection" >> "$newXorgFile"
# Remove the frame buffer drivers
${pkgs.gnused}/bin/sed -i '/-xf86-video-fbdev-/d' $newXorgFile
fi
# Script has completed
exit 0
'';
in
Add the shell script and nvitop to packages
environment = {
systemPackages = with pkgs; [
nvitop # 28Jun24 : cli top for nvidia card
ShellScript-x-config-prepare-script # 27Jun24 : Shell script to provision xorg.conf based on the presence and PCIE address of a specific NVidia card
];
};
Use a systemd service to run the shell script before the graphical.target is reached
systemd = {
services = {
x-config-prepare = {
description = "Prepare X config for boot checking for NVidia card";
documentation = [
"https://discourse.nixos.org/t/services-xserver-videodrivers-does-something-i-didnt-expect/47812"
];
serviceConfig = {
ExecStart = "${ShellScript-x-config-prepare-script}/bin/x-config-prepare-script";
RemainAfterExit = false;
Type = "oneshot";
};
wantedBy = [
"graphical.target" # 27Jun24 : run before graphical.target
];
};
};
};
Blacklist any modules. For me I need to blacklist nouveau and i2c_nvidia_gpu as these both caused me issues.
boot = {
blacklistedKernelModules = [
"i2c_nvidia_gpu" # 28Jun24 : This seems to cause issues for me when Nvidia drivers are loaded
"nouveau" # 29May24 : Block nouveau driver for nVidia card
];
};
Setup the NVidia hardware as if it is going to be used. For me ‘finegrained’ made the eGPU work! Thanks @Solene for this helpful post
hardware = {
nvidia = { # 29May24 : Enable NVidia Hardware Support
modesetting = {
enable = true; # 29May24 : Modesetting is required.
};
nvidiaSettings = true; # 29May24 : Enable the Nvidia settings menu, accessible via `nvidia-settings`.
open = false; # 29May24 : Use the NVidia open source kernel module (not to be confused with nouveau)
package = config.boot.kernelPackages.nvidiaPackages.beta; # 29May24 : Optionally, you may need to select the appropriate driver version for your specific GPU. - stable, beta, production
powerManagement = {
enable = true; # 29May24 : Nvidia power management. Experimental, and can cause sleep/suspend to fail.
finegrained = true; # 29May24 : Fine-grained power management. Turns off GPU when not in use.
};
prime = {
allowExternalGpu = true;
offload = {
enable = true;
};
intelBusId = "PCI:0:2:0"; # 00:02.0 VGA compatible controller: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] (rev 0c)
# The following are replaced by x-config-prepare-script based on the lspci at boot
nvidiaBusId = "PCI:10:0:0"; # 0a:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1) - with dock
#nvidiaBusId = "PCI:4:0:0"; # 04:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1) - without dock
};
};
opengl = { # 29May24 : Enable NVidia Hardware Support
driSupport = true;
driSupport32Bit = true;
enable = true;
};
};
Setup services.xserver to;
- make use of lib, lib.mkAfter and maybe lib.mkForce (I needed it on 23.11 but not on 24.05)
- force x to start with a different config - Thanks @TLATER for this answer
- export xorg.conf to /etc/X11 - Thanks @Atemu for this answer
- load all three drivers. I don’t think this is supported see above.
{ config, lib, pkgs, ... }:
{
services = {
xserver = {
desktopManager = {
xfce.enable = true; # 10Feb24 : Enable the XFCE Desktop Environment
};
displayManager = {
defaultSession = "xfce"; # 10Feb24 : Default Session should be XFCE
lightdm.enable = true; # 10Feb24 : Enable the LightDM Display Manager
xserverArgs = lib.mkAfter [
"-config /tmp/ramdisk/xorg.conf"
"-xkbdir /etc/X11/xkb"
"-logfile /dev/null"
"-verbose 3"
"-nolisten tcp"
"-terminate"
"-logfile /var/log/X.0.log :0"
"-seat seat0"
"-auth /var/run/lightdm/root/:0"
"-nolisten tcp vt7"
"-novtswitch"
];
};
enable = true; # 10Feb24 : Enable the X11 windowing system
exportConfiguration = lib.mkForce true; # 27Jun24 : Put a symlink to the config into /etc/X11/xorg.conf
videoDrivers = [ # 29May24 : Enable NVidia Hardware Support
"modesetting"
"fbdev"
"nvidia"
];
};
};
}
That should be it. Hope it helps someone.