How can the act of merely visiting a Temple and looking at an idol can be haram in any way if you aren’t worshipping said idol?

From what I know about Islam, there shouldn’t be anything wrong with attending a non-Muslim festival as long as you refrain from taking part in any religious ritual that takes place there.