I have a special needs uncle and my whole life I grew up with him being called “removed” and it not being a slur.

It was just a way to describe his mental functioning.

To me it doesn’t have the same impact because I had never heard it used pejoratively until after it was a no-no word.