you are right - the essence of freedom of speech is not that all speech should be allowed all the time and everywhere, but that "speech regulation" has to be decentralized via voluntary contracts between property owners
But.... that seems to be the US approach to any of a number of issues, allow legal freedom and private contracts to restrict. However all too often there seems to be no practical alternative to the restriction for many, so what value does the theoretical freedom really have?