I agree that breaking a production user's experience is as verboten as it gets for agile software development. That's why we have massive test suites and CI infrastructure.
However, you do want to avoid "CI handcuffs", where either because CI is too brittle or because you're simply integrating too many independent projects which, in order to push a change requiring coordinating changes elsewhere, requires massively outsized developer effort in order to push things through with zero CI breakage. This is more of a problem in CI systems with binary metrics like PASS/FAIL where every change that doesn't PASS is rejected.
It's too easy to "wedge" a system like this, where project X's change can't go through without project Y being able to handle the change, and you end up introducing multiple code paths in both projects on a solely temporary basis just in order to keep CI green and happy. Rather than having a zero-tolerance CI failure policy, developers should be allowed to break CI temporarily, so long as they fix it in a timely manner (within an hour or two). Per-developer breakage metrics, to the extent they are needed, should not be in terms of breakage counts but instead breakage durations.
That is, outside of production, it's fine to break stuff and quickly fix it, so long as you don't leave it broken. The big problems are where domain-siloed developers break a zero-tolerance policy because it was necessary to relax it "temporarily", and things stay broken because the policy stays relaxed and cannot be reinstated without sirens going off. Then restoring the CI policy is blocked for everyone by the one guy who knows the AIX quirks for debootstrap or whatever.
Instead, breaking changes should be "allowed" where there is a window to fix the error and still move forward. Only when the window closes without a fix should the breaking change be rolled back (automatically). This line of thinking lends itself to formal, automated policy, but this depends first on judgment and cultural approval.
As it happens, the best CI system I'm aware of for truly distributed, multiple project integration is OpenStack's Zuul: https://github.com/openstack-infra/zuul I'm not sure if it accommodates my prescription above, but if not, they probably have a better idea.
I, uh, don't understand. No sane environment commits straight to prod. You stage to an integration environment. The integration environment can break, no problems. But you only migrate staged dev from the integration environment to prod when all the test pass. Is that not how everyone does things???
Github does it differently. They have "prod" stuff that everyone sees, and "candidate" things which are ALSO deployed from the same codebase (with different enabling flags), and then can do long-term tests (and smaller pull requests) to verify that the New Way (candidate) behaves identically to the Old Way.
Slide 35 [0] of this presentation actually starts the discussion of this exact thing, though Zach has talked about it before. Later, he shows a chart showing the differences as the code in the parallel branch changed over about five hours. (Wow, that's some fast iteration.)
Yes, what you describe is not at all at odds with what I described. If it wasn't clear, I was describing only developer interaction with integration testing per a CI system, with nothing to say about deploying to production other than the token "don't break users" in the first sentence.
I know at least one person working somewhere that has committing to TRUNK allegedly going straight to prod. Thankfully I've managed to avoid such mad places (more by luck than judgement, mind.)
If both projects share a source control system, you can commit a change and update its downstream usages all at once, with no breakage even temporarily. However, this requires source control that scales company-wide. (Git doesn't qualify.)
Not sure why you're bashing on Git here, it certainly does scale to and handle this use case pretty effectively.
This is precisely the problem that submodules solve, much maligned though they are. It's essentially a dependency problem, and we have good answers for those.
Edit: Also I am fairly certain that the kernal experiences these kinds of issues, and is the flagship user of Git.
The Linux kernel is tiny compared to a large company's need for source control. It easily fits on one machine. Think about what you'd need to put all the projects in an entire Linux distro in one source control system; that's more like the scale I'm talking about.
Linus built a tool that suits his needs well but he's not working at the same scale.
Yeah, git-repo-per-project was an unspoken assumption. Also, "upstream and downstream" may not suffice analytically, as many interdependencies in this model do not satisfy DAG, even if they ~should~. Agile projects aren't often built so much as grown.
I think the problem you're assuming here is that dependencies are automatically updated to use the latest version. That sounds like pretty strong coupling in a way that means other problems are lurking somewhere.
Team A should be able to develop without having to consult Team B constantly. That means you have to be mature about deprecating things before you just remove them, but I think that's what the whole article is about.
Yes, there are all the usual ways of deprecating things, migrating the callers, and then finally removing the deprecated methods. This is rather tedious and people tend not to do it for small cleanups like renames. If you can do it in one commit, you can work faster and hopefully get cleaner code.
Of course even if you have atomic source control commits, you don't have atomic deployment so there's still migration to take care of, but it works fine for in-process API's.
However, you do want to avoid "CI handcuffs", where either because CI is too brittle or because you're simply integrating too many independent projects which, in order to push a change requiring coordinating changes elsewhere, requires massively outsized developer effort in order to push things through with zero CI breakage. This is more of a problem in CI systems with binary metrics like PASS/FAIL where every change that doesn't PASS is rejected.
It's too easy to "wedge" a system like this, where project X's change can't go through without project Y being able to handle the change, and you end up introducing multiple code paths in both projects on a solely temporary basis just in order to keep CI green and happy. Rather than having a zero-tolerance CI failure policy, developers should be allowed to break CI temporarily, so long as they fix it in a timely manner (within an hour or two). Per-developer breakage metrics, to the extent they are needed, should not be in terms of breakage counts but instead breakage durations.
That is, outside of production, it's fine to break stuff and quickly fix it, so long as you don't leave it broken. The big problems are where domain-siloed developers break a zero-tolerance policy because it was necessary to relax it "temporarily", and things stay broken because the policy stays relaxed and cannot be reinstated without sirens going off. Then restoring the CI policy is blocked for everyone by the one guy who knows the AIX quirks for debootstrap or whatever.
Instead, breaking changes should be "allowed" where there is a window to fix the error and still move forward. Only when the window closes without a fix should the breaking change be rolled back (automatically). This line of thinking lends itself to formal, automated policy, but this depends first on judgment and cultural approval.
As it happens, the best CI system I'm aware of for truly distributed, multiple project integration is OpenStack's Zuul: https://github.com/openstack-infra/zuul I'm not sure if it accommodates my prescription above, but if not, they probably have a better idea.