Hello `std` - The Nix Flake framework for perfectionists with deadlines!

FWIW, I’ve just started experimenting with std and I really like the approach. For me the biggest value is in it’s establishing standard patterns for the places where there are gaps in, for example, the nix flake output schema, and in allowing the community to start defining more enforceable standards without the main nix project or even std being a bottleneck.

More yapping on the value of convention and standardisation...

Whatever the stack, I love it when there’s a solid, well defined standard for stuff on the periphery of a project, like layout, code style, and similar conventions. That stuff is rarely the focus of the project, so I rather not spend time worrying about it. On the other hand, those periphery decisions also can end up being a real PITA to fix down the road, even if they’re inconsequential early on. It’s even more valuable when taking the first unfamiliar steps with something, when a lack of experience can make the “traps” invisible.

Coming from a Python background, PEP8 and now the black formatter are a godsend. Sure sometimes my personal preferences don’t align 100% with PEP8/black, but the value gained by being consistent with a wider ecosystem far outweighs the minor cost of having to slightly compromise on my preference. Since the standard is set by the larger community, no one on the team has to waste time discussing what style we should adhere to. We can refer to the standard when on-boarding people, but without having to document it from scratch ourselves (writing good documentation and standards is a difficult and under-recognised art IMO!). We also don’t have to reconfigure tooling for our own custom style. So it’s minimal effort for the team, but we get all the benefits that come from consistency, both internally and externally, like it being easier to grok code, since the style is familiar to the eye, and making it easier to spot bugs during review. Equally a good standard doesn’t tie our hands, it doesn’t have to be dogma. If there’s a good reason to deviate from the standard then we still can, and the fact those original decisions for PEP8 are well documented, makes it much easier to work out if the deviation from the spec is justified or comes with a unseen “trap” down the road.

Nix, even in the short time I’ve been using it, seems to have greatly improved standardisation and documented established conventions, especially when flakes came along, but there’s still plenty of gaps. Afaik the only defined output schema for flakes is the one on the wiki which itself just refers to the source for flake check. Nix has a huge potentially to be useful for all kinds of applications, but the schema for flakes doesn’t yet (again afaik) include a standard way to define a library function, or a data set, let alone the huge number of other outputs that it would make sense to standardise.

It seems std goes some way to close the gaps there, and provide a way for the community to start defining other enforceable defacto standards without the main nix project, or any single project, being a bottleneck.

I did really struggle with some of the core language, i.e. the cells, clades, organelles stuff. I think using relatively abstract language is a smart move, especially because it manages to mostly avoid reusing terminology with pre-existing associations, but it means the documentation is a lot more crucial to understanding the basics.

More simple examples would help a lot, and perhaps fleshing out some of the text in the existing “Hello World”. The other examples listed from the wild under the “Introduction” section were a bit too big and complex for me to use in my tentative initial learnings. I think some more super simple, guided examples of example the more general/popular clades would be useful both to bootstrap potential users, and to make it easier to understand the concepts. Maybe starting with data, then going on to nixago then devshells would be a good progression.

Another small thing was that the reference listing for the “Built-in Clades” confused me initially. I should have known better, but before everything clicked, especially for base abstractions, I naively thought those sections were examples of how to use the clades and not the definitions of the clades themselves. As a I say, that was daft of me, since it is listed under the reference section, but when for example I was trying to work out exactly what the devshells clade is for and how it’s used, it seemed like the only relevant link I could click.

Anyway love the project and look forward to using it more! Thanks for the hard work!

1 Like

Thanks for this very well-cast feedback. Indeed, you are already the second person suggesting gradual examples, so that definitly will climb in the priorities list.

I may also reinforce your analysis about exotic naming, the benefits of standardization and the flakes’ output schema bottleneck.

I’d further mention discoverability and forthfollowing subsitution of ever-drifting documentation via the TUI as handy during onboarding.

Ultimately, though, a self-explaining and easy onboarding experience, in short: education, is the most powerful driver of standardization.

Since documentation is not just crucial for std itself, but also for it’s adopters, I’m looking at resucitating styx and weave it into std. I’m not quite satisfied with the vertical (i.e. tooling-centric) aspect of the mdbook toolchain.

std precisely aims to horizontally (i.e. integration-centric) leverage the power of nix-as-fabric / nix-as-lingua-franca across toolings, keeping the domain undisturbed from otherwise blossoming boilerplate grown for holding tool-centric verticals together. A treacherous and innocent-looking boilerplate, that typically kills productivity with Nix for a lot of people. Not always is the goal the journey. :grinning_face_with_smiling_eyes:

Proactive maintenance work on styx started here: github.com/divnix/styx github.com/styx-static/styx

EDIT:

See also: Modernizing `styx`

1 Like

I’d like to see a comparison with https://github.com/hercules-ci/flake-parts

I’m not sure if both share same objectives, but maybe can help to define std better.

This is interesting, OP! I get the feeling that where it’ll really begin to shine is for monorepos, and as someone who has experience working within a sizeable nix-scaffolded monorepo at a previous position I have to admit I don’t hold the hesitance or confusion I’m seeing from a lot of people in this thread. Flakes are a good start for enforcing some form of order, but a strict layout like std seems to have would have been a godsend.

I suspect part of the problem though is that this kind of large nix-scaffolded monorepo experience just isn’t widespread outside of a small handful of organizations and std is one of those kinds of tools where it seems more trouble than it’s worth (and likely is in fact more trouble than it’s worth for most projects) unless you genuinely need it or have been burnt before.

Next time I need to work in a monorepo I’ll keep it in my back pocket for sure though! I look forward to seeing what it grows into.

1 Like

@juaningan, @lambdadog inadvertedly gave the perfect answer to your question about the comparison to flake-parts. Since std deals with integration while eliminating the associated boilerplate, I have been meaning to integrate with flake-parts as soon as I’ve a more concrete need for.

Edit:

In addition:

  • Unlike the module system, Standard doesn’t incentivise destructuring, which is “black magic” and hard for local reasoning.
  • Unlike the module system, Standard doesn’t overly rely on implicit recursivity. That is also very hard to reason about locally and promotes limitless spaghetti code. The availability of the mkForce escape-hatch also breaks configurarion monotonicity and thereby doesn’t enforce proper factoring (over long timespans).
  • Unlike flake-parts, the schema is only one concern addressed. That schema is already enriched ootb with an action library. The schema doesn’t care about the nix CLI’s expectations (that care would be a limiting factor). But you can use std.harvest to break out of the std-native schema in one of the (compatibility) layers of Soil.

Edit2:

Shortly styx will be stdized (I’m almost done). During this refactoring “unwinding recursivity” and dismantling any “global namespaces” was probably the biggest gain in readability and maintainability.

1 Like

There is an interesting discussion going on over at Bob: A build system for microservices powered by Nix - #4 by blaggacao on how to expand std into workflows (as in integration; not as in NIH!).

A couple of doc updates:

1 Like

A little update for those following more closely along:

  • Based on recurring user feedback, Standard has changed semantics.
  • Standard has gained support for just as a gentle on-ramp from the “traditional” task runner world into a Nix based task runner world. This is useful if you want to enable traditional-style contributions to your projects from people who wouldn’t agree to install Nix for a casual contribution (at the expense that they need to manage their environment themselves). — huge shoutout to @jmgilman for implementing this based on his Nixago framework (which landed earlier in Standard).
1 Like

A very exhaustive commented std (WIP) implementation:

https://github.com/input-output-hk/plutus/blob/zeme-iohk/standardisation-2/std/flake.nix

Helps me a lot to finally std make click on me.

1 Like

@juaningan Nice! I didn’t even realize this was public :grinning_face_with_smiling_eyes:. I’ve been planning to write a template project based on these extensive comments (which is nice, because it saves you the big round via the Docs).

I’m also writing another pattern piece for the Docs evolving around the following mental model of CI/CD. Maybe complemented with a Standard GitHub Action … :man_shrugging:

Nice work!

I’m impatient to know how do you std’ize
bitte piece, my use case is
very close to it.

Could be great if a new nixosModule Cell Block Type (formerly Clade)
had an action to navigate interface options a la nixos-search.

EDIT: I guess NixOS module system will be a problem here because its global namespace.

I’m currently on paternity, but afaik this is in the backlog.

I think the semantic precedent that was first set by divnix/DevOS (now divnix/digga) with modules, profiles and profile aggregates (called “suites”) is a good abstraction.

Especially that particular interpretation of “suites” as profile combinators is yet largely unexploited in bitte and conditions the codebase to be sometimes unfortunately really messy.

If / once integration with colmena progresses, Standard can be more natively used for (NixOS) infrastructure-based management.

Currently, we know the “metal” (bitteProfile) and “cloud” (app deployment domain) Cells. Some examples can be seen in the public downstream adopters of Standard & Bitte.

Hey Blagg! I’ve had some ideas about std.

In my view, the goal is to build a portage-like metadata repository of configurations and deployments, which can be used to build, deploy and manage multiple Software-Defined and Network Functions,virtual machines and even bare-metal machines in (singularity-) containers.

Of course, not everything needs to be homogeneous and software-defined from the kernel to the user side or from bootloading to bare-metal networking. The goal is to come as close as possible with plenary power.

Having a canonical way to write conditions for each type of cell would be nice, but may fall short of our idealism.

This could be a good use for the Ultimate Dev Env spec that should be laying around somewhere.

The big goal is to have a single declarative interface to containers and VMs, especially those installed on bare-metal.

This is only part of the pie. We need to handle bootloaders, and, depending on Hardware Support personal interests, to how far all the way to bare metal we can get.

A special case are microservices deployed on bare metal, which we may want to run a hybrid CustomSoftware / CustomResources approach to, but the upper bound is still bare-metal containers.

@DavidRoland I may not grasp to the fullest extend your line of thought and the context of it.

But already, I have a sense that you may enjoy the following:

  • The Aura projects has set sails to blur the current runtime boundaries between pet machines and datacenter workloads. They seem to aim at a classical jail as their namespace primitive. One of the authors after some discussions and back and forth may be currently evaluating Nickel as the lingua franca configuration system for that new runtime. A k8s scheduler-shim is planned. The goal, however, is to replace systemd.

  • Some folks have started to get in touch with some OCI folks and we’ve been talking (just talking the walk as of now) about amending the OCI specification with a Nix image layer type. There are multiple upsides to this, such as quicker startup times, but fundamentally, it brings the Nix distribution model closer to the defacto industry standard for containers. And I also beleive it would disintermediate singluarity ce (at least parts of it), as each hashed resource locator (in the nix store) is typically already signed.

Since aura might use the OCI container spec as artifact interchange format (which I cheer), combining the two might just evolve into some very exciting new mainstream distribution mechanism over time. And one of the Aura authors is actually familiar with Nix and NixOS, which is a promising precursor.

Currently, my focus — as evidenced by the recent pattern pieces in the Standard docs — is on the SDLC and CICDO (CI, CD & Observability) side.

Next, I want to factorize documentation matter and expose close-to-source documentation nuggets onto the Nix repo-spanning connecting tissue via styx. Sort of taking shopify’s backstage heads-on (and partly disintermediate it) with the power of Nix.

I haven’t yet spent a whole lot on SDNs besides some excursions into northbound Yang models, which seemed to be a data description DSL, not unlike Nix/Nickel, and hence probably can be emulated and made be part of the same config lingua franca. A yangModels Cell Block Type could provide a stateless submit Action.

Behold — a paper from La Universidad del Cauca, just around the corner from where I live :slight_smile: :colombia:

http://web.unicauca.edu.co/telematica/?q=node/513

conditions on cells

What kind of conditionals do you have in mind, here?

my bad, i thought you lived on a different planet.

2 Likes

For those interested, the std book has been created to provide a thorough introduction to the framework via a hands-on example project. It’s an evergreen work, so it will receive updates over time, but it’s got enough content to get most people going.

1 Like

That’s really interesting! I wasn’t aware of the Aura project or the effort to add a Nix image layer type to OCI, but that could be really powerful. I’m also really interested in your work on the SDLC and CICD/Observability side. I think Nix is really well-suited for that sort of thing, and I’m excited to see what you come up with.

It sounds like you’re doing some really cutting-edge work here, and I’m really excited to see how it all develops. Keep up the good work Blaggacao, and thanks for sharing your thoughts with us, we’re all excited to see what you come up with next.

Thanks @blaggacao for point Aura.

I thought OAM (Open Application Model) with their only implementation, KubeVela (CUE based), could bring some fresh air to application delivery.

And I’m also wondering how do your CI/CD/O ideas in std will move from runtime to control plane/loop?

1 Like

First, we’ve been exploring the “operable” abstraction as a layer in between “package” and “distribution” (OCI). If the package is already great stuff, it can be tiny. But it grows as a function of bad operational quality of the “package” itself and thereby accomodates important team dynamics over the SDLC.

The planned Standard CI/CD Preset is envisioned to only comprise the Dev environment, since after that we need a more complex state machine to decide on environment propagation. No current CI is really fit for that task. The best Candidate in my opinion for that stage would be zeebe.io (because it can interoperate well with less technical stakeholders), although at work we also have been working on github.com/input-output-hk/cicero for that part of the SDLC focusing on nix-flavord technical stakeholders.

But as for the control plane, all that Std provides is the manifest of the 4th layer of packaging that is handed over to the control plane.

Perspectively, we can have a look at JuJu Charmed Operator Framework for another layer of packaging (“the [still] secret 5th layer”). The advantage is, it doesn’t strictly rely on the control plane’s implementation of distributed state, but brings it’s own and can therefore be ported to different control planes (potentially even systemd).

EDIT: @juaningan you got me hooked on OAM (let someone else bridge it to zeebe :grinning_face_with_smiling_eyes:). If you have an idea or an elegant integration point for CI/CD/O, please comment on that PR: docs: add ci cd pattern piece by blaggacao · Pull Request #123 · divnix/std · GitHub — I guess the 4th layer has to be redefined LOL. And the 5th layer would become somewhat 3bis :rofl:

Support was recently added for generating devcontainers. Usage is straight-forward: supply an existing development shell and get an OCI image back that can optionally be made compatible with VSCode Devcontainers.

{
  vscode = lib.ops.mkDevOCI {
    name = "docker.io/std-vscode";
    tag = "latest";
    devshell = cell.devshells.default;
    vscode = true;
    labels = {
      title = "std-dev";
      version = "0.1.0";
      url = "https://github.com/divnix";
      source = "https://github.com/divnix";
      description = ''
        A prepackaged vscode devcontainer for hacking on std
      '';
    };
}
$ std //_automation/containers/vscode:copy-to-docker
$ devcontainer open .

In the above example, you’ll be dropped into a devcontainer session with the devshell already activated and ready to use. In the near future, the image will be available in a container registry, and it will be possible to use a development shell without having Nix installed locally.

Side note: The container is completely nix-driven and does not rely on using an FHS system like existing tooling.

4 Likes
Hosted by Flying Circus.