packages is just directly translated to nativeBuildInputs, nothing more, nothing less.
The manual covers the theory and difference between the various input types extensively here.
The tl;dr however is, anything you expect to run on the host you are on should go into nativeBuildInputs. That’s why they’re called native, because they are run on the host’s native architecture. They also often include bonus hooks that enable integration in the first place, such as the linking of pkg-config paths your LLM digested from somewhere.
Since things in a shell are virtually always expected to run on the host the shell is built on, it almost never makes sense to use buildInputs in a mkShell call. That’s why packages exists, so that there is an obvious and less confusing argument to put your packages in; people have just been cargo-culting buildInputs since long before that argument existed, so there are still a lot of examples with it floating around.
Most of the time either happens to work, but packages is more correct and enables various hooks to work properly that may otherwise cause you to scratch your head at why stuff isn’t working.
Technically you could choose to separate libraries out into buildInputs, but this in turn only matters for cross-architecture shenanigans, which you’re not going to run into with your everyday shell scenario, and would take significant additional configuration and understanding anyway.
So, in a nutshell, just use packages and forget the other inputs for mkShell exist.