nixos-rebuild rebuilds everything, it just reuses existing build results if they exist. It ignores versions and only cares about .drv differences.
nix-env -u on the other hand doesn’t even know what the source derivation was for any given installed package, it only knows the name/version, so it looks for packages with the same name and a newer version (and has heuristics for dealing with multiple packages that share a name). It then installs just the packages it finds, and leaves everything else untouched. Any package with the same name/version combo is also ignored because it only picks up on newer versions.
Ultimately this comes down to nix-env being imperative instead of declarative. If you want a declarative user package setup, you can use home-manager or you can hack it together with your own scripts. Personally I use this script which is based on one by @LnL7. With that script I just run nix-rebuild and it updates my environment to match a declarative set of packages configured in an overlay.