I have decided to fork Lemmy for a personal project. Being a big fan of nix, I try to use it for anything and everything I do on my dev machine. With that in mind, I figured I could tame some of the complexity in their insane build instructions by doing everything in Nix.
However, as is often the case, this turned out to not be as straightforward as I had hoped.
Here is my current attempt at a flake for the back end build:
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
rust-overlay = {
url = "github:oxalica/rust-overlay";
inputs.nixpkgs.follows = "nixpkgs";
};
# TODO: pull in the Typescript front end as a Submodule
# lemmy-ui = {
# flake = false;
# url = "git+https://github.com/LemInventory/lemmy-ui?submodules=1";
# };
};
outputs = { self, nixpkgs, rust-overlay, flake-utils, ... }:
flake-utils.lib.eachDefaultSystem (system:
let
# overlays = [ (import rust-overlay) ];
overlays = [
rust-overlay.overlays.default
(final: prev: {
rustToolchain = prev.rust-bin.fromRustupToolchainFile ./rust-toolchain.toml;
})
];
lib = pkgs.lib;
cargoToml = builtins.fromTOML (builtins.readFile ./Cargo.toml);
pkgs = import nixpkgs {
inherit system overlays;
};
# rust-toolchain = pkgs.symlinkJoin {
# name = "rust-toolchain";
# paths = [ pkgs.cargo pkgs.cargo-watch pkgs.rust-analyzer pkgs.rustfmt ];
# };
in
with pkgs;
{
packages.default = pkgs.rustPlatform.buildRustPackage {
inherit (cargoToml.package) name;
pname = "lemmy_server";
src = ./.;
cargoLock.lockFile = ./Cargo.lock;
};
devShells.default = mkShell {
buildInputs = [
openssl.dev
postgresql
libiconv
protobuf
pkg-config
];
nativeBuildInputs = [
rustToolchain
# rust-toolchain
pkg-config
openssel.dev
rustfmt
protobuf
];
packages = [
rustToolchain
openssl.dev
pkg-config
cargo-deny
cargo-edit
cargo-watch
rust-analyzer
];
# Environment variables
RUST_BACKTRACE = "1";
# Explicitly set OpenSSL lib and include directories
OPENSSL_LIB_DIR = "${lib.getLib openssl.dev}/lib"; # Note: This is directly accessible
OPENSSL_INCLUDE_DIR = "${openssl.dev}/include";
OPENSSL_DIR = "${openssl.dev.out}";
PROTOC = "${pkgs.protobuf}/bin/protoc";
PROTOC_INCLUDE = "${pkgs.protobuf}/include";
shellHook = ''
# export OPENSSL_LIB_DIR="/nix/store/lvdxawlh51yk1jxx5s0k67mxkil4kq35-openssl-3.0.13/lib"
# export OPENSSL_INCLUDE_DIR="/nix/store/xpz5n8nd9minrr31yw2d63xwg0941xb7-openssl-3.0.13-dev/include"
# export OPENSSL_DIR="/nix/store/lvdxawlh51yk1jxx5s0k67mxkil4kq35-openssl-3.0.13"
# export PROTOC="/nix/store/5g29ind1alcyqa7r820jvhmcpyi5i8pb-protobuf-24.4/bin/protoc"
# export PROTOC_INCLUDE="/nix/store/5g29ind1alcyqa7r820jvhmcpyi5i8pb-protobuf-24.4/include"
export OPENSSL_LIB_DIR="${lib.getLib openssl.dev}/lib"
export OPENSSL_INCLUDE_DIR="${openssl.dev}/include"
export OPENSSL_DIR="${openssl.dev}/out"
export PROTOC="${pkgs.protobuf}/bin/protoc"
export PROTOC_INCLUDE="${pkgs.protobuf}/include"
export RUST_BACKTRACE=1
echo $OPENSSL_LIB_DIR
echo $OPENSSL_INCLUDE_DIR
echo $OPENSSL_DIR
echo $PROTOC
echo $PROTOC_INCLUDE
'';
};
}
);
}
Obviously I was being naive, thinking it would be easy. I know that there are lots of Nix geniuses here, though, so maybe this is an opportunity to learn some advanced Nix.
Here’s the log of the failed build:
Compiling regex v1.10.3
error: failed to run custom build command foropenssl-sys v0.9.101Caused by:
process didn’t exit successfully:/build/60nll1rf369xk7y6734nx0sli5k7nri3-source/target/release/build/openssl-sys-95452a9fe24e59a0/build-script-main(exit status: 101)
— stdout
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
OPENSSL_LIB_DIR unset
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
OPENSSL_INCLUDE_DIR unset
cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
cargo:rerun-if-env-changed=OPENSSL_DIR
OPENSSL_DIR unset
cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG
cargo:rerun-if-env-changed=PKG_CONFIG
cargo:rerun-if-env-changed=OPENSSL_STATIC
cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
cargo:rerun-if-env-changed=PKG_CONFIG_PATH
cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
cargo:rerun-if-env-changed=OPENSSL_STATIC
cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
cargo:rerun-if-env-changed=PKG_CONFIG_PATH
cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
run pkg_config fail: Could not runPKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags openssl
The pkg-config command could not be found.Most likely, you need to install a pkg-config package for your OS.
Tryapt install pkg-config, oryum install pkg-config,
orpkg install pkg-config, orapk add pkgconfigdepending on your distribution.If you’ve already installed it, ensure the pkg-config command is one of the
directories in the PATH environment variable.If you did not expect this build to link to a pre-installed system library,
then check documentation of the openssl-sys crate for an option to
build the library from source, or disable features or dependencies
that require pkg-config.— stderr
thread ‘main’ panicked at /build/cargo-vendor-dir/openssl-sys-0.9.101/build/find_normal.rs:190:5:Could not find directory of OpenSSL installation, and this
-syscrate cannot
proceed without this knowledge. If OpenSSL is installed and this crate had
trouble finding it, you can set theOPENSSL_DIRenvironment variable for the
compilation process.Make sure you also have the development packages of openssl installed.
For example,libssl-devon Ubuntu oropenssl-develon Fedora.If you’re in a situation where you think the directory should be found
automatically, please open a bug at GitHub - sfackler/rust-openssl: OpenSSL bindings for Rust
and include information about your system as well as this message.$HOST = x86_64-unknown-linux-gnu
$TARGET = x86_64-unknown-linux-gnu
openssl-sys = 0.9.101It looks like you’re compiling on Linux and also targeting Linux. Currently this
requires thepkg-configutility to find OpenSSL but unfortunatelypkg-config
could not be found. If you have OpenSSL installed you can likely fix this by
installingpkg-config.note: run with
RUST_BACKTRACE=1environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish…
I realize that some of my issues come from the fact that this build requires two repos (lemmy and lemmy-ui) as submodules rather than one but it looks like I am not even finishing the Rust build because of some dependencies that aren’t being seen. I think the solution is far more complex than I currently know about. So, I brought the issue here to perhaps get some help. If I do end up figuring out how to build this using 100% nix, I will surely submit it as a pull request to the lemmy project.
Here are the scratch-build instructions in the Lemmy documentation.
and here are non-nixy instructions for getting a standalone dev environment working.
Once I get this working, I want to also add an output that builds their Docker containers rather than the super complex scripts they’re currently using. I think they’d get a lot more help on this worthy project if we could solve their devops complexity.