Dear Coder: Please, Don't be a Hero

You surely know this goldie oldie hit from Bonnie Tyler, Holding Out For A Hero? Great piece of music. Mouthfuls of eighties disco stuff, all with a Welsh accent. Unfortunately, we as developers cannot concur with Bonnie when it comes to needing a hero. Here’s why.

Bonnie badly needs a hero and a new haircut — I don’t need the former.

Let’s start by moving on from disco music to technical literature. Do you know the excellent book The Phoenix Project ? I think it’s a must-read for any IT professional. Besides the DevOps manifesto it promotes, it’s also a great piece of storytelling.

I read this book 2 years ago and have been profoundly influenced by its learnings. A key piece it advices against is the occurrence of heroics in the IT world. Heroics occur whenever some developer has to work crazy hours to bring an urgent project to completion because it was due one week ago, or when some Ops engineer has to spend the whole night fixing a buggy production server before the business day starts. I’m sure you’ve been through some of these situations. Maybe you still even recall the adrenaline flowing through your veins at the time.

Coming from a management consulting background and being fond of bravery acts in general, I was initially a bit disappointed to see that in Phoenix heroics were strongly disapproved — although I understood intellectually why it was so. Now, 2 years down the road as a CTO and founder of tech ventures, I understand this condemnation of heroics too well.

So here’s a deeper analysis : why should one avoid heroics at all cost in their tech organisation, or simply for themselves as developer?

Why heroics are bad for tech organisations

Because heroics are damn too risky

If you’ve ever been responsible for a piece of production software, I guess you despise taking risks that might jeopardise the running of that application. That is, unless these risks are clearly quantified as the missing crumbs between 99.99% and 100%. All the rest is out-of-boundaries stuff and should be avoided, because this unpredictability hampers your health and lifestyle (bugs are known to increase your level of cortisol, the stress hormone).

Mr Hero = Mr Risk ? Quite so.

The reason we software maintainers are able to limit unpredictability is because we follow strict processes, carefully orchestrated from specs to QA to deploy. This might sound constraining at first, but once mastered it proves extremely rewarding in terms of developers’ productivity. Now, what happens when heroes pop out of the woods to fence off the bugs dragons? They’ll happily diverge from these carefully crafted procedures. They’ll take shortcuts to face the beast quicker.

Hero story: from time to time at Seraphin, we encounter failures in the middle of complex Interactors that manage our insurance business processes. Our recovery procedures can require manual intervention to reach the desired end-state. Ideally, whenever a process fails, one should assess where exactly it got stale, create an as-generic-as-possible rake correction routine, load it on the servers, then execute it remotely using Capistrano. This way everything is version-controlled and auditable. What too often happens however is that we SSH directly on the servers, open the Rails console, and fix stuff by hand directly (after the occasional test-run on a local copy of the production DB). Fast and easy, huh? No! Bad heroics!

Although such tricks can prove very rewarding when done successfully (and boost the ego of the dev that fixed the bug, pleasing the customer), they can wreak havoc when things go wrong. When you venture away from processes, you lose your safety nets.

Because heroes are paying for their manager’s failures

My stand is that as a manager you should always prioritise the well-being of your team. For this, you need to make sure that the following is clear between you and fellow developers when you’re assigning them a feature:

• Full context of the feature to be developed (where, when, by whom it will be used, how critical it is, what are potential further developments to it)
• Foresight on downstream implementation complexities and watchouts
• Potential implementation shortcuts and trade-offs, along with non-negotiables in terms of scope

Failing to establish this common base of understanding leads to unpleasant consequences, especially if your team members are full of motivation. They might well end up working well into the night on things that you would happily have de-prioritised as a manager or product owner, but that you failed to communicate about. Fatigue and loss of motivation ensue.

Everyone makes mistakes.

Hero story: at Seraphin, we’ve recently decided to break our Rails monolith in a series of micro-services largely supported by serverless AWS Lambda functions. This is proving a great choice, but a key limitation of Lambdas is that it’s not always possible to pack them with all libraries and binaries we’d like. Initially unaware of this, I asked a developer in my team to package a proxy to an SOAP service in a Python Lambda. The job turned out to be a tough one, because Python requires a C compiled binary to handle SOAP calls — something you cannot easily port to Lambda. The dev thought it was important to be able to handle any SOAP call and thus to have this library. He also thought this issue was urgent enough that it required from him to work late in the evening on it. Worth a medal but clearly, I failed to communicate to him (1) that only a few defined SOAP calls would be required, making it possible to use a simple XML construction instead of a library and (2) that this whole thing, being a refacto of our codebase, was by no means an emergency requiring red-eye work.

I now always overemphasise to my team the context of their work and the complexities I foresee in the implementation. The more knowledge they have, the more they are able to take the right trade-offs to save their lifestyle when confronted with technical challenges. All in the team know that I don’t expect them to be heroes — that would be an indicator of my failure.

Because heroes are paying for their company’s failures

Similarly, good-willing programmers are also sometimes forced to put heroic tricks because of structural failures in their company. This is especially true when it comes to the following critical points:

• Ability of the business to clearly articulate and communicate their needs
• Ability of the company’s project manager to buffer sufficient time for testing before any major release

It’s so unjust that developers get to mend the pieces when the buffer is all eaten…

Again here, the manager has an important role to play to safeguard the team’s well-being against company weaknesses. Developers are indeed particularly exposed to organisational failures in the course of projects as they reside at the right end of the value chain and will not be provided with the same buffer as their business peers. It is important for developers to realise this and take a stand to reinforce the organisation’s project management capabilities in a cross-teams effort, instead of deploying heroics after heroics to meet deadlines. This is true even in Agile setups.

Hero story: let me ask you a question here. Out of the devs and business people in your company, who are the ones spending the most on office-delivery pizzas during times of harsh projects? Yeah. Right. Developers foot the highest bill! And believe me, that’s not because programmers are more found of junk food — but rather because in my experience they’re more often forced to put all-nighters to make up for delays accumulated over the course of a project. Call it the pizza syndrome: biz people eat up the buffer, developers eat pizza at night in the office as a consequence ;)

Because going hero means going solo

An issue with heroes is that they’re not really concerned with team work. More often than not, they’re lone rangers looking for challenges to achieve. This is of course inspiring, but tech work is first and foremost about team work. Knowledge gathered by one developer over the course of an implementation that is not shared subsequently with the team is lost knowledge! Tech is all about co-learning, peer reviews, and mutual challenging. It’s about preventing anyone from becoming indispensable because they’re the only one with a specific piece of knowledge.

Me? Goin’ solo?

One should hence refrain from tackling too big challenges on their own just for the sake of achieving badass stuff. And remember, there are other places for heroes to shine: innovation challenges, exploration of new technologies…it is possible to frame moments where everyone’s quest for individual recognition is satisfied.

Hero story: in the early days of Seraphin, I was the only in-house developer. It was thus sometimes extremely tempting for me to discard the documentation process altogether on certain tasks: I had very little time and so much to do, and there wasn’t anyone out there to read the doc anyway. I remember in particular the harsh process of migrating our entire production infrastructure from OVH to AWS (a move we never regretted, of course). This took me 2 nights — the first attempt failed because of a stupid DNS limitation at OVH’s side not clearly indicated in their doc. Well, although the whole thing was quite stressful and done absurdly late at night (to minimise visible downtime), I still took the time to document every single action I took in the course of the migration. Now, 2 years down the road, this document is still being used in our team as a recipe to setup new modules of our infrastructure. I resisted the temptation of going fully solo at the time, and this proved super helpful afterwards.

A closing thought

Truth be told, it’s not always easy to resist the temptation of going hero. At the end of the day, this is all about preferring collaboration and development of team dynamics over personal achievements. When understood by all, these principles help building more powerful teams with more meaningful interactions. I hope you found interest in them!

Justice League: heroes teaming together — maybe the ultimate role models for dev teams?