GPG prompts for passwords for unrelated keys

[MrKaplan-lw]

For some sops-encrypted files the correct PGP key is picked for decryption right away.
For other files I'm getting prompts to enter the password for unrelated PGP keys that are neither listed in .sops.yaml creation rules nor able to decrypt the files.

Is there a way to prevent this?
It gets very annoying when you have multiple encrypted private keys that aren't unlocked and amplified even more by having multiple encrypted files.

It may relate to the file type, it consistently decrypts a yaml file with the correct key right away, but various other files like .env files always try other keys first, prompting me to enter a password for keys I don't need or cancelling that for it to try the next key.