Unexpected nix and ripgrep interaction

Help
1

Preamble

Sorry for a (very) long post. I wanted to capture potentially relevant details. You can skip most of it unless you’re interested in details (which I’ve marked with Details heading).

I’m mostly posting this to see if some of you can reproduce this and if you have any ideas what’s going on or if you have any pointers regarding how I can troubleshoot this further.

I’m not sure if this is specific to nix, ripgrep or a combination of the two. If you have an idea how I can reproduce this on non-NixOS systems (given it depends on nix commands), let me know and I’ll try / open an issue in ripgrep repo if I can reproduce.

The issue

When filtering the output of nix registry list, for some reason ripgrep is not outputting the match every time. Example that reproduces the issue on my end:

$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always; done
1

2

3
global flake:agda github:agda/agda
4

5
global flake:agda github:agda/agda
6
global flake:agda github:agda/agda
7

8

9
global flake:agda github:agda/agda

Details

Using NixOS unstable (rev 6e51c97f1c849efdfd4f3b78a4870e6aa2da4198 in flake.lock, 298add347c2bbce14020fcb54051f517c391196b latest as of now)

$ rg --version
ripgrep 13.0.0
-SIMD -AVX (compiled)
+SIMD -AVX (runtime)

Reproducible in text-only terminal (i.e. Ctrl+Alt+F1) and multiple GUI terminals (console, gnome-terminal and tilix).

This happens with other nix commands as well, e.g.:

$ for j in {1..9}; do echo $j; nix flake metadata nixpkgs|rg URL; done
1

Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
2

Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
3
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
4
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
5

Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
6
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable

7
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
8
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b
9
Resolved URL:  github:NixOS/nixpkgs/nixpkgs-unstable
Locked URL:    github:NixOS/nixpkgs/298add347c2bbce14020fcb54051f517c391196b

Thing’s I’ve tried so far that work fine:

  • Using no colors with ripgrep 
$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=never; done
1
global flake:agda github:agda/agda
2
global flake:agda github:agda/agda
3
global flake:agda github:agda/agda
4
global flake:agda github:agda/agda
5
global flake:agda github:agda/agda
6
global flake:agda github:agda/agda
7
global flake:agda github:agda/agda
8
global flake:agda github:agda/agda
9
global flake:agda github:agda/agda
  • Using grep instead of ripgrep 
$ for j in {1..9}; do echo $j; nix registry list|grep -m1 global --color=always; done
1
global flake:agda github:agda/agda
2
global flake:agda github:agda/agda
3
global flake:agda github:agda/agda
4
global flake:agda github:agda/agda
5
global flake:agda github:agda/agda
6
global flake:agda github:agda/agda
7
global flake:agda github:agda/agda
8
global flake:agda github:agda/agda
9
global flake:agda github:agda/agda
  • Saving the output to a file first
$ nix registry list > j; for j in {1..9}; do echo $j; cat j|rg -m1 global --color=always; done
1
global flake:agda github:agda/agda
2
global flake:agda github:agda/agda
3
global flake:agda github:agda/agda
4
global flake:agda github:agda/agda
5
global flake:agda github:agda/agda
6
global flake:agda github:agda/agda
7
global flake:agda github:agda/agda
8
global flake:agda github:agda/agda
9
global flake:agda github:agda/agda
  • Redirecting stderr to stdout
$ for j in {1..9}; do echo $j; nix registry list 2>&1|rg -m1 global --color=always; done
1
global flake:agda github:agda/agda
2
global flake:agda github:agda/agda
3
global flake:agda github:agda/agda
4
global flake:agda github:agda/agda
5
global flake:agda github:agda/agda
6
global flake:agda github:agda/agda
7
global flake:agda github:agda/agda
8
global flake:agda github:agda/agda
9
global flake:agda github:agda/agda
  • Using cat after ripgrep 
$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always|cat; done
1
global flake:agda github:agda/agda
2
global flake:agda github:agda/agda
3
global flake:agda github:agda/agda
4
global flake:agda github:agda/agda
5
global flake:agda github:agda/agda
6
global flake:agda github:agda/agda
7
global flake:agda github:agda/agda
8
global flake:agda github:agda/agda
9
global flake:agda github:agda/agda

It looks like this is purely an output issue, since:

$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always --files-without-match; done
1
2
3
4
5
6
7
8
9


$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always --files-with-matches; done
1

2
<stdin>
3

4
<stdin>
5
<stdin>
6
<stdin>
7
<stdin>
8

9
<stdin>


$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always --files-with-matches|cat; done
1
<stdin>
2
<stdin>
3
<stdin>
4
<stdin>
5
<stdin>
6
<stdin>
7
<stdin>
8
<stdin>
9
<stdin>


$ for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always -c; done
1
1
2
1
3

4
1
5

6

7

8

9



 for j in {1..9}; do echo $j; nix registry list|rg -m1 global --color=always -c|cat; done
1
1
2
1
3
1
4
1
5
1
6
1
7
1
8
1
9
1
2

Try running ripgrep in faketty and tee the result into a file. You may find some clues in the exact terminal escape sequences flying around.

1 Like
3

Unfortunately, just using faketty causes it to work (in the same way piping through cat does in one of the examples I pasted above):

 for j in {1..9}; do echo $j; nix registry list|rg --color=always nixpkgs; done
1

2

3

4

5
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
6

7

8

9


$ for j in {1..9}; do echo $j; nix registry list|faketty rg --color=always nixpkgs; done
1
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
2
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
3
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
4
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
5
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
6
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
7
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
8
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable
9
global flake:nixpkgs github:NixOS/nixpkgs/nixpkgs-unstable

Are you able to reproduce this on your end? Just wanted to confirm it’s not something specific to my setup.

4

I am able to reproduce it. And I think I know what’s happening:

$ (faketty nix registry list >/dev/null) |& hexdump -C
00000000  0d 1b 5b 30 6d 1b 5b 4b  0d 1b 5b 4b 0d 1b 5b 30  |..[0m.[K..[K..[0|
00000010  6d 1b 5b 4b 0d 1b 5b 4b  0d 1b 5b 30 6d 1b 5b 4b  |m.[K..[K..[0m.[K|
00000020  0d 1b 5b 4b 0d 1b 5b 30  6d 1b 5b 4b 0d 1b 5b 4b  |..[K..[0m.[K..[K|
00000030  0d 1b 5b 30 6d 1b 5b 4b  0d 1b 5b 4b 0d 1b 5b 30  |..[0m.[K..[K..[0|
00000040  6d 1b 5b 4b 0d 1b 5b 4b  0d 1b 5b 30 6d 1b 5b 4b  |m.[K..[K..[0m.[K|
00000050  0d 1b 5b 4b 0d 1b 5b 30  6d 1b 5b 4b 0d 1b 5b 4b  |..[K..[0m.[K..[K|
00000060

nix’s fancy statusline system may be interleaving with the output, even though it really shouldn’t even be active for a command like this, and ripgrep is just adding enough delay that the race condition shows itself.

2 Likes
5

Ah OK makes sense - so basically nix is outputting some escape sequences to stderr even though it’s piped. Actually, it probably isn’t the escape sequences that are the problem here - the \r at the beginning is probably causing issues with ripgrep’s output.

Thanks for debugging this. I’ve created nix CLI outputting ANSI escape sequences to stderr when piped · Issue #7558 · NixOS/nix · GitHub

1 Like