Hello `std` - A DevOps framewok for the SDLC with the power of Nix and Flakes

That is one way of looking at it. Perhaps a specific example would help. If we look at the oci-images organelle declared here. This tells us that if any one of these cells in the cells directory contain either an “oci-images.nix” or an “oci-images/default.nix” in their top level, that the file should be a function (as you say) taking { inputs, cell }: that returns a set that must contain only nix2container built OCI images as values.

And std is kinda like glue in that it calls this function and exports the resulting values, also checking that they are the delcared type.

cell is to reference to the current cell so you can access the other organelles defined in them, say if you define some useful library functions. You can access any other cell in the flake from the special inputs.cells, or any cells from other flakes from inputs.

1 Like

This is one example, the implementation is a bit buried, I don’t want to entice you to dig, but TL;DR; it’s pretty much the same as plumber-cd/terraform-backend-git.

1 Like

I’m currently gathering feedback for an entrypoint-writer with passthru mkOCI & mkDebugOCI: imp: add standardized writeShellEntrypoint by blaggacao · Pull Request #86 · divnix/std · GitHub

If you can spare a friendly review, constructive feedback, I’d really appreciate it.

1 Like

For what it’s worth, Nixago-support just landed in std:

https://github.com/divnix/std/pull/97

Huge thank you to @jmgilman — was great working together on this!

TL;DR;

The std wrapper for numtide/devshell aquired a new nixago option:

{
      nixago = [
        (std.nixago.conform {configData = {inherit (inputs) cells;};})
        cell.nixago.treefmt
        cell.nixago.editorconfig
        cell.nixago.mdbook
        std.nixago.lefthook
        std.nixago.adrgen
      ];
}
2 Likes

Yeah i still don’t understand any of these use case but i suppose it is just a widely different use case niche for nix than the one i live in. :slight_smile:

FWIW, I’ve just started experimenting with std and I really like the approach. For me the biggest value is in it’s establishing standard patterns for the places where there are gaps in, for example, the nix flake output schema, and in allowing the community to start defining more enforceable standards without the main nix project or even std being a bottleneck.

More yapping on the value of convention and standardisation...

Whatever the stack, I love it when there’s a solid, well defined standard for stuff on the periphery of a project, like layout, code style, and similar conventions. That stuff is rarely the focus of the project, so I rather not spend time worrying about it. On the other hand, those periphery decisions also can end up being a real PITA to fix down the road, even if they’re inconsequential early on. It’s even more valuable when taking the first unfamiliar steps with something, when a lack of experience can make the “traps” invisible.

Coming from a Python background, PEP8 and now the black formatter are a godsend. Sure sometimes my personal preferences don’t align 100% with PEP8/black, but the value gained by being consistent with a wider ecosystem far outweighs the minor cost of having to slightly compromise on my preference. Since the standard is set by the larger community, no one on the team has to waste time discussing what style we should adhere to. We can refer to the standard when on-boarding people, but without having to document it from scratch ourselves (writing good documentation and standards is a difficult and under-recognised art IMO!). We also don’t have to reconfigure tooling for our own custom style. So it’s minimal effort for the team, but we get all the benefits that come from consistency, both internally and externally, like it being easier to grok code, since the style is familiar to the eye, and making it easier to spot bugs during review. Equally a good standard doesn’t tie our hands, it doesn’t have to be dogma. If there’s a good reason to deviate from the standard then we still can, and the fact those original decisions for PEP8 are well documented, makes it much easier to work out if the deviation from the spec is justified or comes with a unseen “trap” down the road.

Nix, even in the short time I’ve been using it, seems to have greatly improved standardisation and documented established conventions, especially when flakes came along, but there’s still plenty of gaps. Afaik the only defined output schema for flakes is the one on the wiki which itself just refers to the source for flake check. Nix has a huge potentially to be useful for all kinds of applications, but the schema for flakes doesn’t yet (again afaik) include a standard way to define a library function, or a data set, let alone the huge number of other outputs that it would make sense to standardise.

It seems std goes some way to close the gaps there, and provide a way for the community to start defining other enforceable defacto standards without the main nix project, or any single project, being a bottleneck.

I did really struggle with some of the core language, i.e. the cells, clades, organelles stuff. I think using relatively abstract language is a smart move, especially because it manages to mostly avoid reusing terminology with pre-existing associations, but it means the documentation is a lot more crucial to understanding the basics.

More simple examples would help a lot, and perhaps fleshing out some of the text in the existing “Hello World”. The other examples listed from the wild under the “Introduction” section were a bit too big and complex for me to use in my tentative initial learnings. I think some more super simple, guided examples of example the more general/popular clades would be useful both to bootstrap potential users, and to make it easier to understand the concepts. Maybe starting with data, then going on to nixago then devshells would be a good progression.

Another small thing was that the reference listing for the “Built-in Clades” confused me initially. I should have known better, but before everything clicked, especially for base abstractions, I naively thought those sections were examples of how to use the clades and not the definitions of the clades themselves. As a I say, that was daft of me, since it is listed under the reference section, but when for example I was trying to work out exactly what the devshells clade is for and how it’s used, it seemed like the only relevant link I could click.

Anyway love the project and look forward to using it more! Thanks for the hard work!

1 Like

Thanks for this very well-cast feedback. Indeed, you are already the second person suggesting gradual examples, so that definitly will climb in the priorities list.

I may also reinforce your analysis about exotic naming, the benefits of standardization and the flakes’ output schema bottleneck.

I’d further mention discoverability and forthfollowing subsitution of ever-drifting documentation via the TUI as handy during onboarding.

Ultimately, though, a self-explaining and easy onboarding experience, in short: education, is the most powerful driver of standardization.

Since documentation is not just crucial for std itself, but also for it’s adopters, I’m looking at resucitating styx and weave it into std. I’m not quite satisfied with the vertical (i.e. tooling-centric) aspect of the mdbook toolchain.

std precisely aims to horizontally (i.e. integration-centric) leverage the power of nix-as-fabric / nix-as-lingua-franca across toolings, keeping the domain undisturbed from otherwise blossoming boilerplate grown for holding tool-centric verticals together. A treacherous and innocent-looking boilerplate, that typically kills productivity with Nix for a lot of people. Not always is the goal the journey. :grinning_face_with_smiling_eyes:

Proactive maintenance work on styx started here: GitHub - divnix/styx: Static site generator in Nix expression language. (fork maintained for https://github.com/divnix/std) GitHub - styx-static/styx: Static site generator in Nix expression language.

EDIT:

See also: Modernizing `styx`

1 Like

I’d like to see a comparison with GitHub - hercules-ci/flake-parts: ❄️ Simplify Nix Flakes with the module system

I’m not sure if both share same objectives, but maybe can help to define std better.

This is interesting, OP! I get the feeling that where it’ll really begin to shine is for monorepos, and as someone who has experience working within a sizeable nix-scaffolded monorepo at a previous position I have to admit I don’t hold the hesitance or confusion I’m seeing from a lot of people in this thread. Flakes are a good start for enforcing some form of order, but a strict layout like std seems to have would have been a godsend.

I suspect part of the problem though is that this kind of large nix-scaffolded monorepo experience just isn’t widespread outside of a small handful of organizations and std is one of those kinds of tools where it seems more trouble than it’s worth (and likely is in fact more trouble than it’s worth for most projects) unless you genuinely need it or have been burnt before.

Next time I need to work in a monorepo I’ll keep it in my back pocket for sure though! I look forward to seeing what it grows into.

1 Like

@juaningan, @lambdadog inadvertedly gave the perfect answer to your question about the comparison to flake-parts. Since std deals with integration while eliminating the associated boilerplate, I have been meaning to integrate with flake-parts as soon as I’ve a more concrete need for.

Edit:

In addition:

  • Unlike the module system, Standard doesn’t incentivise destructuring, which is “black magic” and hard for local reasoning.
  • Unlike the module system, Standard doesn’t overly rely on implicit recursivity. That is also very hard to reason about locally and promotes limitless spaghetti code. The availability of the mkForce escape-hatch also breaks configurarion monotonicity and thereby doesn’t enforce proper factoring (over long timespans).
  • Unlike flake-parts, the schema is only one concern addressed. That schema is already enriched ootb with an action library. The schema doesn’t care about the nix CLI’s expectations (that care would be a limiting factor). But you can use std.harvest to break out of the std-native schema in one of the (compatibility) layers of Soil.

Edit2:

Shortly styx will be stdized (I’m almost done). During this refactoring “unwinding recursivity” and dismantling any “global namespaces” was probably the biggest gain in readability and maintainability.

1 Like

There is an interesting discussion going on over at Bob: A build system for microservices powered by Nix - #4 by blaggacao on how to expand std into workflows (as in integration; not as in NIH!).

A couple of doc updates:

1 Like

A little update for those following more closely along:

  • Based on recurring user feedback, Standard has changed semantics.
  • Standard has gained support for just as a gentle on-ramp from the “traditional” task runner world into a Nix based task runner world. This is useful if you want to enable traditional-style contributions to your projects from people who wouldn’t agree to install Nix for a casual contribution (at the expense that they need to manage their environment themselves). — huge shoutout to @jmgilman for implementing this based on his Nixago framework (which landed earlier in Standard).
1 Like

A very exhaustive commented std (WIP) implementation:

https://github.com/input-output-hk/plutus/blob/zeme-iohk/standardisation-2/std/flake.nix

Helps me a lot to finally std make click on me.

1 Like

@juaningan Nice! I didn’t even realize this was public :grinning_face_with_smiling_eyes:. I’ve been planning to write a template project based on these extensive comments (which is nice, because it saves you the big round via the Docs).

I’m also writing another pattern piece for the Docs evolving around the following mental model of CI/CD. Maybe complemented with a Standard GitHub Action … :man_shrugging:

Nice work!

I’m impatient to know how do you std’ize
bitte piece, my use case is
very close to it.

Could be great if a new nixosModule Cell Block Type (formerly Clade)
had an action to navigate interface options a la nixos-search.

EDIT: I guess NixOS module system will be a problem here because its global namespace.

I’m currently on paternity, but afaik this is in the backlog.

I think the semantic precedent that was first set by divnix/DevOS (now divnix/digga) with modules, profiles and profile aggregates (called “suites”) is a good abstraction.

Especially that particular interpretation of “suites” as profile combinators is yet largely unexploited in bitte and conditions the codebase to be sometimes unfortunately really messy.

If / once integration with colmena progresses, Standard can be more natively used for (NixOS) infrastructure-based management.

Currently, we know the “metal” (bitteProfile) and “cloud” (app deployment domain) Cells. Some examples can be seen in the public downstream adopters of Standard & Bitte.

Hey Blagg! I’ve had some ideas about std.

In my view, the goal is to build a portage-like metadata repository of configurations and deployments, which can be used to build, deploy and manage multiple Software-Defined and Network Functions,virtual machines and even bare-metal machines in (singularity-) containers.

Of course, not everything needs to be homogeneous and software-defined from the kernel to the user side or from bootloading to bare-metal networking. The goal is to come as close as possible with plenary power.

Having a canonical way to write conditions for each type of cell would be nice, but may fall short of our idealism.

This could be a good use for the Ultimate Dev Env spec that should be laying around somewhere.

The big goal is to have a single declarative interface to containers and VMs, especially those installed on bare-metal.

This is only part of the pie. We need to handle bootloaders, and, depending on Hardware Support personal interests, to how far all the way to bare metal we can get.

A special case are microservices deployed on bare metal, which we may want to run a hybrid CustomSoftware / CustomResources approach to, but the upper bound is still bare-metal containers.

@DavidRoland I may not grasp to the fullest extend your line of thought and the context of it.

But already, I have a sense that you may enjoy the following:

  • The Aura projects has set sails to blur the current runtime boundaries between pet machines and datacenter workloads. They seem to aim at a classical jail as their namespace primitive. One of the authors after some discussions and back and forth may be currently evaluating Nickel as the lingua franca configuration system for that new runtime. A k8s scheduler-shim is planned. The goal, however, is to replace systemd.

  • Some folks have started to get in touch with some OCI folks and we’ve been talking (just talking the walk as of now) about amending the OCI specification with a Nix image layer type. There are multiple upsides to this, such as quicker startup times, but fundamentally, it brings the Nix distribution model closer to the defacto industry standard for containers. And I also beleive it would disintermediate singluarity ce (at least parts of it), as each hashed resource locator (in the nix store) is typically already signed.

Since aura might use the OCI container spec as artifact interchange format (which I cheer), combining the two might just evolve into some very exciting new mainstream distribution mechanism over time. And one of the Aura authors is actually familiar with Nix and NixOS, which is a promising precursor.

Currently, my focus — as evidenced by the recent pattern pieces in the Standard docs — is on the SDLC and CICDO (CI, CD & Observability) side.

Next, I want to factorize documentation matter and expose close-to-source documentation nuggets onto the Nix repo-spanning connecting tissue via styx. Sort of taking shopify’s backstage heads-on (and partly disintermediate it) with the power of Nix.

I haven’t yet spent a whole lot on SDNs besides some excursions into northbound Yang models, which seemed to be a data description DSL, not unlike Nix/Nickel, and hence probably can be emulated and made be part of the same config lingua franca. A yangModels Cell Block Type could provide a stateless submit Action.

Behold — a paper from La Universidad del Cauca, just around the corner from where I live :slight_smile: :colombia:

http://web.unicauca.edu.co/telematica/?q=node/513

conditions on cells

What kind of conditionals do you have in mind, here?

1 Like

my bad, i thought you lived on a different planet.

3 Likes