I would guess that the letter frequency, digram and trigram frequencies etc are quite skewed compared to random. Just by taking that into account, you would crack an average password much faster than you suggest by trying the most likely passwords first. There are plenty of already cracked passwords to draw statistics from.

That in addition to traditional dictionary attacks.