Introducing Crystal Forge: Compliance-First NixOS Fleet Management (Very Early MVP)

Hello! I wanted to share a project I’ve been working on for the past several months and get some feedback from the community. This is Crystal Forge and I want to be upfront: it’s super early stage. At best, it’s a proof of concept. These past few months have been largely me learning Rust and the nuances of the Nix CLI.

What I’m Building Toward

The long-term vision is to make NixOS easier to use in compliance-heavy environments (government, banking, defense, etc.) than traditional solutions like RHEL. I’ve put together some slides that give an overview of my plans: https://crystalforge.us

Why This Matters

The TL;DR: Nix gives us deterministic, cryptographically-verifiable knowledge of exactly what’s on every deployed system. This means we can:

  • More easily verify compliance
  • Deploy compliant configurations with confidence
  • Know precisely when ALL systems are patched
  • Track the complete provenance of every system state

Crystal Forge just puts a nice bow on things and stores it all in a database to make it easier when it comes to do your audits.

What Actually Works Today

Right now Crystal Forge is really just for homelabbers and tinkerers, but here’s what’s functional:

A Build and deploy framework - Basic monitoring and tracking of systems across multiple Flakes/Git repos

Deployment policies - Simple policies like “always latest” or “manual,” with the groundwork for more complex, extensible policies. Currently they just verify Crystal Forge is enabled, but this can be extended to verify STIGs or other compliance requirements

CVE scanning - Using Vulnix to scan all systems and store results (will integrate this into deployment policies next)

Basic Grafana dashboards - Rudimentary but functional for monitoring. Eventually I want a proper web interface, but Grafana works for now

What’s Coming

I have proof-of-concept STIG generation functions (example-here) that will require attestation for why certain controls are disabled. The goal is to keep all accreditation metadata in a single, verifiable place and output it to formats like OSCAL.

Technical Notes

This is my first Rust project, so there’s definitely some learning-curve and AI slop scattered throughout the code The builder services use systemd-run for resource limiting. With proper configuration, you should be able to reliably build even the heaviest systems (Electron, Firefox, etc.). I have a Ryzen 9 7950X3D and am able to pretty reliably build Electron and Firefox without problem. I’ve implemented an extensive integration test suite, though there are still gotchas I’m working through (like actually building inside NixOS test VMs. If anyone has ideas, I’m all ears!)

Links

Crystal Forge repo: https://gitlab.com/crystal-forge/crystal-forge

My dotfiles (to see it in action): https://gitlab.com/usmcamp0811/dotfiles

Next Steps

My immediate priority is a big refactor to clean up the architecture and consolidate some of the experimental implementations. After that, I’ll work on CVE visualization and integrating CVE data into deployment policies. I’m not a project manager, a security person, or a professional Rust developer – I’m figuring this all out as I go. I’ve had some input from friends, but it’s mostly been me hacking away at this. There’s a ton I know I need to do, and I’m completely open to thoughts, feedback, and collaboration. If this sounds interesting to you, I’d love to hear your ideas!

13 Likes

This is neat!

What things so far were sticking points during development? What things were surprisingly not so terrible?

Thanks! So much to say here… I intend to write a blog post about my experiences so far…

So first thing before I started I didn’t realize I was going to have to build everything to do Vulnix scanning, so that immediately lead me to expanding scope. Which building was/is harder than it seems. I knew I wanted to not have CF crashing just because a config needed to build Firefox or Electron, so I had to limit resources to the builds, but I also didn’t want to limit everything to one job because I felt that would be too slow. So long story short I went through a number of iterations where I was playing with different build processes. Had to learn more about systemd-run and ran across nix-fast-build which I ended up realizing I was already doing a lot of that, but then just decided to use nix-eval-jobs to speed up the eval steps.

This was all on top of just learning Rust and working through some different database schemas as I have different ideas or learn something new about Nix that causes me to need to make changes. Like I started using the .drv paths for my primary key but later realized it was better to just use the store path.

Eventually I’d like to use the Rust implementation of Nix directly inline, but from what I could gather thats still pretty early days too, to I just went with Nix proper.

1 Like