The root cause is quite literally profiteering by the private sector, combined with rigged zoning that makes it cheaper to build inefficiently.

Government or other not-for-profit housing of some kind (such as renter co-ops, which would be even better to have more of) are the only thing that make solid sense when considering available options, because anything else is routinely being bought up so it can be flipped for profit or rented at high rates few can afford.