Signal has been pretty extensively audited and the source code is entirely available so I’m pretty sure we’d have heard about it by now. It should be impossible, if Signal don’t have your private keys… but there’s still an amount of trust that they haven’t been compromised and had a malicious binary uploaded, or their encryption hasn’t been broken, etc.

If you’re worried you can read the source code yourself: https://github.com/signalapp