The Reaper Will Come
I was originally writing an article that looks at the time and costs for engineering decisions. Addressing both the immediate and long team effects.It will follow on the heels of this one. However, I noticed that I needed to introduce the idea of feedback and its longterm effects. Hence, this article was born.
My mother used to say, “It is not a question of if the reaper will come but when he will come and how much will he reap.” Yes, she was amazing, but this one quote has always guided my life and career.
Guided by mother’s tail of the reaper, I approach any decision in software engineering I ask the question: “How much will this decision cost us and how much will it cost to reverse it”.
People constantly tell me this is a cynical view. In reality, it is a pragmatic approach to couching each decision in consequences.
Originally I used to pull effects from my bum but then I found “interaction diagrams”. These allowed me to discuss cause/effect with teams1. Not perfect but a visualization to your decisions and their effect.
This is an unfortunate example of how local optimizations caused the demise of a promising startup.
I was VP of Engineering for a startup in a highly volatile space. I was hired to revive a team going into a first round of funding. I catalogued the problems but got the go ahead when their live site went down for 3 days during a board meeting.
My team spent a month implementing CICD with full automated regression tests. We got to 4 to 8 releases a day with zero defects. Note we used new features to drive these changes.
The end to end build to stage with tests took less than ten minutes. The regression tests making solid releases gave the company a distinct advantage in a volatile market.
We had a solid, evolving platform for a few months. At a staff meeting two team leads clamored for long lived GIT branches for “reasons” at a staff meeting. I am using “reasons” here derogatorily because they really had none: it was basically an appeal to emotion. It was essential I isolated how much it will cost and try to see if we got value from it.
Apparently, the teams were worried about code that was not correct getting into the branch. I asked,
“Why would you check in broken code?”
Of course I got some guffaws and wails claiming I was attacking them. I mean, honestly, you should ever check in broken code? How is it possibly a good thing to share knowingly damaged code? So I switched to how much will this cost us.
I did the sequence diagram that showed the result of the long lived branches and that because of merge/integration times people would tend to go longer between pulls from and pushes to master which created a feed back loop making integrations more painful. There are actually a few feed back loops including big merges makes pulls from master longer because it breaks things, and multiple side branches because the long lived branch has become unstable dilates the full integration more.
I have been in companies where the integration cycle from these long lived branches takes months(!)… yah months.
Internally my main fear was that the reason for long lived branches was a lack of development hygiene on the part of the mid managers, tech leads and developers. Long lived branches just hide an obfuscate the lack of discipline.
Fast forward 4 weeks: All progress stopped (deployment/integration times were now weeks long). developer asked why it was so painful to merge when we used to do it effortlessly.
My only consolation, since the company was going down hard, was I got a kick ass “I told you so” moment when I brought up the original interaction diagram to answer the developer’s question and his response was (bless him), “Why did we do that?”
I think the disillusion that longed lived branches comes from Linus and his rightful demand that master branch of the Linux kernel should be pristine.
I will lean on Linus and say all checkins should be pristine. Indeed all code should be as pristine as possible at all times (generally I use15 to thirty minute intervals). The real question is:
“How do you define pristine?”
In the Linux case, Linus had an amazing ability to keep the entire kernel in his head, so coxs 55de that passed his scrutiny is pristine.
For us mere mortals, the only z
We should demand ‘pristine’ branches, but they should be
