Namaka - Snapshot testing for Nix

Announcements
1

10 Likes
2

It’ll be a true pleasure to embrace this tool next :partying_face:

Just as it was to embrace Haumea.

For people who wonder what is this good for?

My (current) condensed conclusion would be: it takes a snapshot of your (e.g.) NixOS resolved value lattice * and helps you make important parts of your configuration an invariant.

Please educate me if I got this wrong. :grinning_face_with_smiling_eyes:

* Why are we commonly still reasoning about config data in terms of NixOS? :stuck_out_tongue_winking_eye: It’s just the beginning.

1 Like
3

I don’t get it. What does this?

5 Likes
4

A tool for snapshot testing.

Means you have a list of tests, which are just function calls or values and a name.

You do not specify the result, as you do with traditional tests, but do a first run, and trust the results, the tool will persist those for you and on later runs it will compare to persisted values.

12 Likes
5

What is the snapshot part of this testing?

You do not specify the result, as you do with traditional tests, but do a first run, and trust the results, the tool will persist those for you and on later runs it will compare to persisted values.
@NobbZ

Is this not just cached testing instead?

6

The snapshots live in the tests/_snapshots directory of the repo

I wasn’t able to find what cached testing is, could you elaborate on that?
I also found this strategy being called approval tests, is that what you are looking for?

7

I think I am just completely confused about what the software does in general, I am going to go look at the repo in more depth.

8

I think @NobbZ explained it pretty well. Look up “snapshot testing” on google, it’s a pretty common terminology in software. You should also look up “snapshot testing vs unit testing”.

3 Likes
9

I was able to write a test on a complex output datastructure (several hundred LoC) that would have otherwise taken hours to craft in just about 15 mins.

That’s an absolutely awesome capability!

3 Likes
10

I guess this is similar to golden testing, as well? If you want another term to look up.

3 Likes
11

TIL about golden testing, namaka is indeed very much the same.

1 Like
12

namaka v0.1.1 was just released which adds a lot of the missing documentation, including a quick start guide, and a section to (briefly) explain what snapshot testing is.

This update also adds templates that hopefully makes it easier to get started with namaka

nix flake init -t github:nix-community/namaka
# or
nix flake init -t github:nix-community/namaka#minimal

Features

  • load: accept src
  • cli: allow running commands other than nix flake check

Changes

  • load: deprecate flake and dir

Fixes

  • load: fix json format when values are implicitly casted
  • canonicalize path before running nix flake check
3 Likes
13

namaka v0.2.0 has been released with support for configuration and some quality of life changes!

Breaking Changes

  • namaka review now runs nix eval .#checks by default instead of nix flake check. You can override this behavior by editing namaka.toml or using the --cmd flag.
  • When provided a directory, namaka now changes the working directory to it, instead of modifying the nix commands it runs. This can make a difference if you specify both the directory and --cmd.

Features

  • Local configuration with namaka.toml
  • namaka clean to remove unused and pending snapshots
  • Add subflake template for using namaka in a subflake

Improvements

  • namaka review now runs faster by default, since the default command has been changed to nix eval .#checks.