Well, for any major fitness difference, evolution could happen considerably more quickly than millions of years. A 10% difference would be noticeable in only hundreds of years, and a 1% difference in thousands.