I think that not being "productive" is scorned simply because most people are forced into involuntary wage-slave "productivity", and those people hate others who escape it.

But it's also scorned by those who employ the wage-slaves. The profits of "productivity" accrues mostly to the employers, and those who eschew this state of things must therefore be condemned.