Counterpoint: Knowledge of the infrastructure can be used in social engineering attacks, e.g. to increase the likelihood of success for password spearphishing.
If people with access to your infrastructure can be spearphished for passwords, I would venture to say that you're probably doing other stuff wrong that needs to be fixed first.
