The second group I worked with that was earnestly interested in mature testing developed the 5/8ths rule.
To move a test one level down the pyramid, it takes about 5x as many tests. But the tests run 8 times as fast. So moving a test down takes more than 35% off the run time, and it fails the build minutes sooner. If you drop it down two levels it's 60% off the run time.
Interesting enough on its own, but maintaining those tests after one requirements change, plus the cost of rewriting them in the first place, is less work than the cost of maintaining the original tests. We didn't come up with a number for this but the difference was measured in man-days and missed deadlines about once a month, and we were convinced by the evidence.
I also agree with both your 'braindead' comment and your 80% estimate. The big payoffs come between 75% and 85% and above 85% you start getting artifacts. That 'data' distract more than it helps.
Yup. I think one big issue is that a E2E or an integration test is useful on its own, while a single unit test is almost totally worthless. You don't have confidence of anything until at least 50% (and at 80% you have almost perfect confidence).
So when people get started, especially on an old code base, they feel its pointless and doesn't pay off. Can't blame them, I suppose.
Good that you bring up build time. I forgot to mention that. We have repos with thousands of tests where the whole suite runs in <1 minute and gives us very high confidence (actually the only other tests we run on that repo are visual regression tests for CSS, and even E2E tests don't catch those issues...). During that time I'm watching other teams waiting 20 minutes on their integration test suite. Nope nope nope.
To move a test one level down the pyramid, it takes about 5x as many tests. But the tests run 8 times as fast. So moving a test down takes more than 35% off the run time, and it fails the build minutes sooner. If you drop it down two levels it's 60% off the run time.
Interesting enough on its own, but maintaining those tests after one requirements change, plus the cost of rewriting them in the first place, is less work than the cost of maintaining the original tests. We didn't come up with a number for this but the difference was measured in man-days and missed deadlines about once a month, and we were convinced by the evidence.
I also agree with both your 'braindead' comment and your 80% estimate. The big payoffs come between 75% and 85% and above 85% you start getting artifacts. That 'data' distract more than it helps.