Ahh that makes sense, with more people involved it’s harder to go by. I use Fossify Contacts, and it has an option to save contacts but to make them invisible to other apps. Perhaps that might help you somehow, I’m not sure

Actually, contacts access isn’t strictly necessary! I run it with the permission disabled, and although it complains regularly, it still works. I don’t have any issues with having to remember people’s numbers either since they’re recognisable from the profile pictures and because I talk to the same few people usually