The issue is that you’re changing the ecosystems and environments so much that all those eons of evolution are simply lost. The only other times this happens is during natural catastrophes. Sure, in the long run this allows new life forms to take the old ones places, but it’s still a massive loss of diversity and evolutionary knowledge - and unnecessary suffering for millions of living beings.

When species compete for a habitat, they rarely destroy it - and those species that do either don’t survive for long, or they wipe out large swaths. We’re actively killing almost anything in our habitats, and destroying them for almost all previous species.