Hello `std` - A DevOps framewok for the SDLC with the power of Nix and Flakes

For those interested, the std book has been created to provide a thorough introduction to the framework via a hands-on example project. It’s an evergreen work, so it will receive updates over time, but it’s got enough content to get most people going.

1 Like

That’s really interesting! I wasn’t aware of the Aura project or the effort to add a Nix image layer type to OCI, but that could be really powerful. I’m also really interested in your work on the SDLC and CICD/Observability side. I think Nix is really well-suited for that sort of thing, and I’m excited to see what you come up with.

It sounds like you’re doing some really cutting-edge work here, and I’m really excited to see how it all develops. Keep up the good work Blaggacao, and thanks for sharing your thoughts with us, we’re all excited to see what you come up with next.

Thanks @blaggacao for point Aura.

I thought OAM (Open Application Model) with their only implementation, KubeVela (CUE based), could bring some fresh air to application delivery.

And I’m also wondering how do your CI/CD/O ideas in std will move from runtime to control plane/loop?

1 Like

First, we’ve been exploring the “operable” abstraction as a layer in between “package” and “distribution” (OCI). If the package is already great stuff, it can be tiny. But it grows as a function of bad operational quality of the “package” itself and thereby accomodates important team dynamics over the SDLC.

The planned Standard CI/CD Preset is envisioned to only comprise the Dev environment, since after that we need a more complex state machine to decide on environment propagation. No current CI is really fit for that task. The best Candidate in my opinion for that stage would be zeebe.io (because it can interoperate well with less technical stakeholders), although at work we also have been working on GitHub - input-output-hk/cicero: event-driven automation for Nomad for that part of the SDLC focusing on nix-flavord technical stakeholders.

But as for the control plane, all that Std provides is the manifest of the 4th layer of packaging that is handed over to the control plane.

Perspectively, we can have a look at JuJu Charmed Operator Framework for another layer of packaging (“the [still] secret 5th layer”). The advantage is, it doesn’t strictly rely on the control plane’s implementation of distributed state, but brings it’s own and can therefore be ported to different control planes (potentially even systemd).

EDIT: @juaningan you got me hooked on OAM (let someone else bridge it to zeebe :grinning_face_with_smiling_eyes:). If you have an idea or an elegant integration point for CI/CD/O, please comment on that PR: docs: add ci cd pattern piece by blaggacao · Pull Request #123 · divnix/std · GitHub — I guess the 4th layer has to be redefined LOL. And the 5th layer would become somewhat 3bis :rofl:

Support was recently added for generating devcontainers. Usage is straight-forward: supply an existing development shell and get an OCI image back that can optionally be made compatible with VSCode Devcontainers.

{
  vscode = lib.ops.mkDevOCI {
    name = "docker.io/std-vscode";
    tag = "latest";
    devshell = cell.devshells.default;
    vscode = true;
    labels = {
      title = "std-dev";
      version = "0.1.0";
      url = "https://github.com/divnix";
      source = "https://github.com/divnix";
      description = ''
        A prepackaged vscode devcontainer for hacking on std
      '';
    };
}
$ std //_automation/containers/vscode:copy-to-docker
$ devcontainer open .

In the above example, you’ll be dropped into a devcontainer session with the devshell already activated and ready to use. In the near future, the image will be available in a container registry, and it will be possible to use a development shell without having Nix installed locally.

Side note: The container is completely nix-driven and does not rely on using an FHS system like existing tooling.

4 Likes

I’ve started to prompt some phrases from the Standard docs to an AI text2image (pixray). Just wanted to share. I’m stunned.


“Nix is marvel to ones and cruelty to others.”

image


“Lazy configuration tree”

1665803405924_20221014222313


“Nix has superpowers”

16658023307958672982907227321733_20221014222359


“The Nix Operator persona …”

image

For the ones interested, I just managed to produce a decent integration betweem Standard and Colmena over here, thanks to a new colmena upstream interface that allows to relax on the schema (of which kost parts would have unused added complexity in this use case that would have to be muted in userspace). Now it’s muted by design in the evaluator amd yhe user interface is super simple (and there’s only one!):

i really want to understand std and hive… i can pretty much checkout any config and understand it… but with hive… i stare on it and don’t get it :frowning:
and i really want to understand it…

1 Like

Well, it’s also just in the making. I’m not even using it for myself, so it might be a bit early.

But I can still say something about it.

First, std hasn’t been made with NixOS configurations in mind. This might be a potential use-case, though, and hence, I’m trying out stuff in divnix/hive.

You’ve got to understand how std does imports as a prerequisite. And probably familiarize with its few conventions.

Then, in the cellBlocks secion of divnix/hive we kind of see a pattern that should be familiar to divnix/digga / divnix/DevOS users: the distinction in modules, profiles & suites.

Then, we also have the “final” configurations. So in a way it’s 4 layers: module → profile → suite → configuration. (If “4 layers” triggers somebody in another context, please send me a :sunflower: :grinning_face_with_smiling_eyes: :wave: )
Ignore (devshells "devshells"), it’s for the dev environment for the flake itself: rather standard Standard :grinning_face_with_smiling_eyes: .

So what you’ve been looking at so far is the hive’s output type system that is intelligently leveraged by the importer. Just as any type system, this gives you a decent, but abstract overview over the semantic organization of this particular repo.

Next, the “# soil”. The soil are all additional variadic arguments (after the first one) to the growOn function, and not by coincidence is the soil located down (towards the earth). It’s the “soil”, after all. Easy to reference, easy to find (across all std repos). It is designed to host all the compatibility things, where some tooling X (including the nix CLI’s defaults) expects things. So if you know std and you see a “#soil”, it tells you something about tooling schemata (or just “legacy” stuff).

So in divnix/hive/flake.nix#soil, you see adapters for colmena, nixos-generators & home-manager (in the making, so maybe not yet). The functions have absolutely unrelated and undocumented fantasynames for a purpose: they are prototypes and by not choosing a final naming, they are less likely to slip into a mental corsette, already. Other than maybe making the reader dizzy lol! :grinning_face_with_smiling_eyes: :champagne:

So what does make-{mead,honey,moonshine}.nix do? They harvest the cells in the schematic ways that each of the tools expect. This is where you read unlimited variations of boilerplate in those other repositories, that look “easier” at first glance. Yeah, each time, you’d be looking mostly at some variation of boilerplate in such a prime estate like a flake.nix, such as if “using Nix” was about “writing boilerplate”… :man_artist:

So if you drop the mental presumption of the primacy of the tooling schema (read: “tool centric”), then what you’re left with is a perfectly “problem” or “domain”-centric (and typed, sic!) file organization with a compatibility layer of “#soil”. And the best: the problem domain is yours to explore. No opinion whatsoever does std have (as opposed to, for example, the Nix CLI).

At this point: nix repl / :lf . / outputs.<TAB> tells the rest of the story (and makes the small bits of magic that std actually really does intuitively experiencable). I’m afraid, that’s probably the “cost” of learning Standard.

std TUI/CLI? Won’t be helpful in divnix/hive, we have quite well-established vertical tooling (colmena/nixos-generator/home-manager). No need to wrap them in a horizontal one. The use case for the CLI/TUI is rather the one of a “nix” service provider to a “non-nix” configuration consumer.

2 Likes

@luxus just fiddled this together over the weekend, so maybe the NixOS use case becomes clearer with this example for you and others: feat(lina2358): add initial [working] configuration by blaggacao · Pull Request #2 · divnix/hive · GitHub

Some notes:

  • No nixosProfiles/nixosSuites/nixosModules (not needed for a simple config)
  • Why nixosConfigurations if we have colmenaConfigurations? → no real good answer, maybe just because colmena is an external evaluator while nixosConfigurations can be explored in the repl, natively. Maybe compat with nixos-generators’ CLI for what it’s worth.
  • makes use of the new nix-community/disko’s module integration (very WIP)
  • Still separates hardwareProfile so that there’s not going to be a layer violation and flipping the metal shall not change the system config (there are still a couple of layer violations in this config, but it’s good enough for now)
  • Others I forgot? → maybe Better WIFI UX on the non-graphical installer. I really got to like ifwifi over the weekend.

Still, at this point using std for NixOS is off-label use and probably will always be or maybe an off-shoot project might take that space, cause entering the fearless NixOS support pool is not necessarily something that I would want to invest in near term. It would also likely start to completely overshadow the fledgling Nix (not NixOS!) DevOps use-case & patterns.

But if it was to support NixOS more closely, it would take out a few “full repository cycles” as to when it comes to passing instances of nixpkgs (pkgs) down to the module system and settle on one single way (aligned across colmena, nixos & home-manager).

1 Like

Quick Update: We’ve started to modularize some components of Standard and make them independently useful:

A little brush-up on the readme. Hope this is a better appetizer than what’s been there before…

And there is also an integration with https://flake.parts live now:

https://flake.parts/options/std.html

  • You can use flake-parts with std as described there.
  • Or you can use std with flake-parts by simply invoking flake-part’s mkFlake function in one of your layers of “Soil”.
3 Likes

This is ongoing work and lacks context. Still I want to tease out some feedback.

1 Like

I found std while searching if OAM had any nix-related stuff going on, and only found this open issue:

As nix newbie and fretting that nix would be too complex and overkill, std and the way that it is documented really made things click for me, and now I’m sold on Nix et al :blush:

The context of my search is a project where I want to integrate Wasmcloud, that has a complex custom build environment, and uses wadm (wasmCloud Application Deployment Manager) based on OAM. I too feel that this open standard and nix can be complementary.