It isn’t required that they be an MP, only that they lead the party, have the approval of the governor, and have the confidence of Parliament. In this case, though, they can’t vote on any laws, since they haven’t been elected. The usual solution is for an elected MP of the party in a riding that is strong for that party to step down, triggering a by-election with the PM runs in. This has happened before, and will doubtless happen again.