A small update on the subject:
I found this post, which makes me think that I’m a victim of the same kind of problem. Before deciding whether the submodule instance is a bookmark or directory, it only checks that it’s either an attrset, function or a valid path and picks the first submodule. I could probably override the check function of the two types to check that it’s both an attrset and that they include certain non-null attributes (like url), but I don’t know what to do about function/path values. I’m also left wondering why checking the non-null attrs isn’t the default behaviour. I’m guessing there’s probably a very valid reason somewhere in the internal workings of nix.
However, I’m be changing my aim towards a list based structure, because I forgot that attrsets don’t keep order. So in my case, there’s no need for this stuff to work just yet. But feel free to continue the thread if you’re stuck at the same problem.
EDIT: I found types.addCheck, which seems quite useful for this usecase!