RobG3D

In my previous post about technical debt, I explained how modern definitions of technical debt are harmful. Now I turn my attention to equally harmful metaphors.

Viktoras Makauskas made the following metaphor in a comment on my last post. This is a pretty perfect stand-in for metaphors I’ve read in other articles that harmfully define technical debt.

Imagine your car gets a strange rattle. You go to your mechanic and he says, “it’s your exhaust pipe holder, you need to replace it, but it’s gonna take a while to order a part and ship it, so just park your car here and come back in a week”. You say “no, I have this weekend trip planned, is there something we can do now?”. They say “yeah, we’ll put a strap on it meanwhile, just drive a little more careful and it should hold, but make sure to come back and do a proper fix”. Mechanic charges you now, and then a bit later.

This seems sensible on first read. But upon closer inspection, it’s quite clear the roles here are totally wrong*:

• The mechanic is the programmer (the role of the “expert”). Well, a mechanic may or may not see your car ever again. They do not have a vested interest in your choice. A mechanic’s relationship to a car is totally different from a programmer’s relationship to code.
• “You” are the “business” (the role of the “stakeholder”). The metaphor assumes that if you are irresponsible, it only impacts you (it’s your car, your money, your time). This is a problem. A programmer is impacted by business decisions in a way a mechanic is not impacted by whether you fix your car now or later.

This isn’t a simple language problem. It is a fundamental misunderstanding of roles that is naive to the way software development works. Programmers will be the primary sufferers of technical debt. Eventually the business will suffer with a slower pace of innovation and development and higher turnover. But well before that, programmers will be fixing (and refixing) obscure bugs, will bristle under management that tells them to go faster, will be working extra hours to try to improve things, and will eventually burn out. The business will only suffer once real damage has been done to a programming team, and many have given up.

This is why control of technical debt must be in the hands of programmers. Definitions or metaphors that urge otherwise are actively harmful.

Let me close by pointing out I’m just repeating what Ward Cunningham has already written about the original technical debt metaphor. The article ends with:

A lot of bloggers at least have explained the debt metaphor and confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later and thinking that that was the primary source of debt.
I’m never in favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of a problem even if that understanding is partial.

Thanks Ward.

* There are also a couple other problems with this metaphor. First, if “you” and the mechanic are the same person, and responsible for both business and implementation? In that case, there’s no need for a metaphor at all. Second, what happens if the exhaust fails? Do you become stranded? Does the car catch fire? What’s presented here is a false choice between a “correct” solution (replacement) or a “sloppy” solution (strapping it on). Why not rent a car? If there’s no responsible-but-relatively-cheap decision (there almost always is!), it’s still never acceptable to make an irresponsible decision.

David Smith over at baleful.net makes some interesting points about the length of most interviews:

So mathematically, you will most likely get the highest confidence interval with: 1) Resume screen, 2) Phone interview, 3) In-person interviews 1-3. From the above, this should represent about 50% of the total causes, but should produce 91% of the total effect. Adding additional interview steps after that 91% brings only incremental improvement at best and backslide at worst.

He makes an extremely compelling argument, and I encourage you to read the entire piece. That said, I still prefer a full day of interviews as both the interviewer and interviewee.

The interviewee angle is easy. I enjoy interviews. I like to dig into my potential employer. I want to grill your second-string players. I want to hear how junior people feel treated. I want as much information as possible before making my choice. But I know this is just me, and people who are less comfortable with interviews probably prefer shorter ones. I also admit I don’t think I’ve learned anything in the second half of a day of interviewing that would have made me turn down a job. But I have learned things that helped me in my job once hired.

The benefits of full-day interviews for the interviewers is much more complex. There are several factors:

• We have diverse backgrounds and expertise, and each group brings a unique perspective. Candidate postmortems are not dominated by the same couple interviewers.
• I want to give as many people experience interviewing as possible. I consider it an important skill. Limiting things to three in-person interviews means the interviewers are all “musts” and I don’t get to experiment at the periphery with groups or combinations.
• People want to be a part of the process. I’ve personally felt frustrated when left out of the process, and I know I’ve frustrated others when I’ve left them out.

For a developer role, I want them to meet with at least: founders, ops, lead developer, two developers, myself. We’re at an absolute minimum of 7. That is with a narrow set of views, without inexperienced interviewers, and leaving good people out. What am I supposed to do?

• For starters, the interview process should be more transparent and collaborative. Ask the interviewer if they want a full day, two half days, morning or afternoon, etc.
• No group lunches. I’ve never gotten useful feedback from a group lunch. Keep it down to one or two people. A candidate just doesn’t want to embarrass themselves, so they just shut up, and side conversations dominate.
• Avoid solo interviews. I used to hope to solo interview everyone. But over time, I’ve found that pairing on interviews enhances the benefits listed above. There are still times I will want a solo interview, but in general I will pair.
• Cut the crap. Interviewers should state their name and role. Don’t bother with your history unless asked. Don’t ask questions that are answered by a resume. Instead of “tell us about yourself” how about “tell us what you’re looking for”.
• Keep a schedule. Some people are very bad at managing time. If someone isn’t done, too bad, keep things moving. They will eventually learn how to keep interviews to their allotted time.

Thanks to David for the insightful post. I’ll continue to keep full-day interviews, but we’ll definitely change some things up.

Ben Sandofsky wrote a post about why QA departments are still necessary, specifically with regards to mobile app development. He makes a good point: mobile apps create a distribution bottleneck that makes very rapid iteration impossible. I agree, and this is a good angle to think about. I would have been happy with an article focused on this.

Ben is clearly a talented guy but this post was insane. In a literal sense. It is a rant for anti-Agile curmudgeons at best, and would leave me questioning the experiences of anyone that thinks this way at worst.

Websites ship embarrassing bugs all the time. They get away with it because they didn’t ship it to all users. You roll-out everything out to 1% of users, and watch your graphs. If things look good, slowly roll out to 100%.

The idea that this is this sort of incremental rollout is ubiquitous amongst web developers is crazy. It requires infrastructure, code designed to support split testing, experienced operations engineers, robust monitoring, a disciplined process, and more. The institutions with this sort of sophistication all have strong automated testing environments. Which brings me to my next issue:

I think automated testing accelerates development, but I haven’t seen a direct correlation between testing and quality. On projects with massive, high quality test coverage, I’ve seen just as many bugs slip through as projects with zero coverage.

This is the software equivalent to climate change denial. Where does this experience come from? I am not sure I’d be able to find a single developer who would corroborate this. Oh, right:

Tell a game developer you don’t need [QA], they’ll tell you you’re nuts.

The game industry is full of these folks who believe what they are doing is such an untestable snowflake. Unsurprisingly, games have historically been the buggiest software around. Never, ever look at game development as an example of how to do QA right. Not just automated testing, but manual QA too.

…a great QA team is far from a bunch of monkeys clicking buttons all day.

Game development has a hallmark technique of hiring masses of QA people and have massive layoffs at the end of projects. There is an entire website dedicated to tales of horror from QA people. It makes The Daily WTF look like paradise.

Take the unicorn of “two week release cycles.” As you build infrastructure for faster releases, simple code becomes unwieldy. Tasks that should take hours take weeks.

What does this even mean? There are endless apps on two week release cycles. I am confused how building infrastructure for faster iterations ends up adding complexity to simple code or tasks.

Disciplined development is a lost art.

You could make this argument when we moved away from punch cards. But the idea that success in mobile apps is achieved through discipline, but success on the web can be achieved by recklessness, is beyond baseless. It’s downright insulting.

I consider it a tragedy that, when faced with the reality of App Store distribution bottlenecks, Ben’s answer is to go back to the process of yesteryear and throw out the lessons we’ve learned. Why not invent new ways of building in quality? New ways of iterating on apps faster? There are so many interesting problems to solve.

Finally, Ben cautions:

Today, any web developer who wants to stay employed has learned to build apps. If web companies want to remain relevant, they’ll have to do the same.

I have a better warning. Don’t throw away the incredible advances we’ve made over the last decade. Don’t downplay the success and rate of innovation in web development as something that doesn’t apply. Don’t throw away the universal “good idea-edness” of automated testing. Don’t rely on a separate department to enforce quality. Don’t stop looking for ways to make development better.

Uncle Bob, who I consider my favorite programming writer, had a post a few weeks ago titled “Thorns around the Gold“. In it he describes how writing tests for your core functionality first can be harmful. Instead, Uncle Bob prefers to probe for “thorns” around the “gold” first.

I shy away from any tests that are close to the core functionality until I have completely surrounded the problem with passing tests that describe everything but the core functionality. Then, and only then, do I go get The Gold.

I haven’t been doing TDD for nearly as long as Uncle Bob but I was shocked to read this. I’ve always learned and taught that you should create positive tests first, and only need as many negative tests as you feel are warranted. While you may not grab the gold immediately, you at least step towards the gold. How many thorns you expose is a judgement call. In Python, most people don’t even bother validating for None inputs, and instead just let things raise (or not). Of course, this depends on your users. For libraries limited to one internal application, I wouldn’t “probe many hedges.” For open source libraries, I validate pretty aggressively.

Of particular interest was this:

I often find that if all the ancillary behaviors are in place before I approach the core functionality, then the core functionality is much easier to implement.

I always thought you should only program what you need and no more. It seems very strange to assume the ancillary behaviors will be needed. It seems like a violation of YAGNI.

I have been trying to reconcile Uncle Bob’s advice here, and the TDD best practices I’ve learned and developed. But I cannot. Either I’ve been receiving and giving years of bad advice, or Uncle Bob has made a rare mistake.

From this post by Rands, titled “The QA Mindset”:

At the current gig, there’s no QA department. […]

My concern is that the absence of QA is the absence of a champion for aspects of software development that everyone agrees are important, but often no one is willing to own. Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective. Traditionally, they are known as the folks who break things, who find bugs, but QA’s role is far more important. It’s not that QA can discover what is wrong, they intimately understand what is right and they unfailingly strive to push the product in that direction.

I believe these are humans you want in the building.

At my current job, we don’t have a QA department either. And like Rands, I wasn’t comfortable at first. I’ve worked on teams without QA, but an entire company without a QA Department? I’ve certainly had questions about the use of a QA department, but does that mean they are a bad idea?

Yes, and this line in Rands’ defense is why:

Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective.

I am a staunch believer of “building quality in.” Every bug that slips out is a failure of your development process. The way to higher quality is not to find, or fix, more bugs. It’s to avoid them in the first place.

If you rely on QA to champion unit testing, automation, bug tracking, and quality metrics, your development process is lacking its most important tools and measures to improving quality. Quality can’t be imposed by QA, it must grow out of enabled and engaged development teams.

I have a saying: “Don’t hire to fix a problem.” If you have a quality problem, hiring a QA department isn’t going to fix it. You instead hide the systematic problems that cause quality issues in the first place.

This is not to say “the QA mindset” isn’t valuable. It is. One of my best hires was Bjorgvin Reynisson, who was a Test Engineer at Nokia and I hired as a QA Engineer at CCP. He was embedded with the graphics engine team and he helped them develop extensive automated correctness and performance testing systems. He worked with them to recognized holes in their process and test coverage. He helped with tracking issues and increasing quality. This is the “QA Mindset” I treasure, and this type of person is invaluable to development teams. Bjorgvin unlocked a latent “culture of quality” in the team he was a part of.

I contrast this “QA Mindset” with the “QA Department Mindset“. The QA Department Mindset has two damning characteristics. First, it is innately adversarial, as Rands notes.

Yes, there is often professional conflict between the teams. Yes, I often had to gather conflicting parties together and explain[…]

Second, it is by definition a separate department, which creates obstacles to better integrating engineering and QA.

Bjorgvin should be spending time with his teammates and the rest of the developers figuring out how to improve the entire development process. He should not be spending time with other QA personnel focused on QA functions. When I was Technical Director for EVE Online, I made sure there were minimal discussions gated by job title. Talk of a trade went on in Communities of Practice, which were open to all. Sometimes this didn’t happen, and those times were mistakes.

Like Rands says:

Yes, we actually have the same goal: rigorously confirming whether or not the product is great.

If that’s the case, QA should not be broken out into a separate department. QA should be working side by side, reporting into the same people, measured by the same success metrics, contributing to the holistic success of an entire product.

I love the QA Mindset. It’s tragic that having a QA Mindset gets confused with having a QA Department.

From a wonderful post by Matt Williams about the type of business he is looking for:

A Business Manifesto
We are uncovering better ways of running a business and helping others do it.
Through this work we have come to value:
– People and interactions over profits and prestige
– Quality service over quantity of service
– Customer relationships over contract negotiation
– Flexibility over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

In a nutshell I want to work for a company which values people — both inside and out of the company. I want to work where people strive to do things right.

When I go home, I want to be able to look in the face of my daughter and not have to make excuses for the work that I do and the effect it has on others.

Sums things up nicely (and definitely what we aspire to at Cozy, by the way we’re hiring).

It is a reason I left the video games industry. I wanted to use my skills to do something I felt was more constructive.

But more than that, I was amazed and frustrated with how the industry was run (almost as bad as films). Mass layoffs even on successful projects. Over-managed projects that go on for 4, 5, 6 years and are cancelled. Creating an exploitative product in order to milk a customer base. Huge budgets, huge marketing, appeals to lowest common denominators (often sexual). There are good companies but the business models are so insane that you can be around for 10 years and fold tomorrow.

I was once told in a performance review that “perception is reality.” I was infuriated, and the words stuck in my mind as the most toxic thing a manager could say to an employee. I have avoided writing about it, but the “This American Life” episode about Carmen Segarra’s recordings at the Fed has inspired me to change my mind. Here’s the relevant section, emphasis mine:

Jake Bernstein: Carmen says this wasn’t an isolated incident. In December– not even two months into her job– a business line specialist came to Carmen and told her that her minutes from a key meeting with Goldman executives were wrong, that people didn’t say some of the things Carmen noted in the minutes. The business line specialists wanted her to change them. Carmen didn’t.

That same day, Carmen was called into the office of a guy named Mike Silva. Silva had worked at the Fed for almost 20 years. He was now the senior Fed official stationed inside Goldman. What Mike Silva said to Carmen made her very uncomfortable. She scribbled notes as he talked to her.

Carmen Segarra: I mean, even looking at my own meeting minutes, I see that the handwriting is like nervous handwriting. It’s like you can tell. He started off by talking about he wanted to give me some mentoring feedback. And then he started talking about the importance of credibility. And he said, you know, credibility at the Fed is about subtleties and about perceptions as opposed to reality.

Well shit, if that doesn’t sound familiar. Here I was, doing work that was by all measures extremely successful, yet pulled into a feedback meeting to be told “perception is reality.”

Let me tell you what “perception is reality” means, and why you should plan on leaving your job the moment you hear it:

The arbitrary opinions of your manager’s manager defines your situation. And they don’t like what you’re doing.

Your manager may be well-meaning (mine was, as was Mike Silva), but at the point you get this “perception is reality” bullshit, you can be sure there’s nothing that they are going to do to help you. Someone above them has taken notice, your manager has probably heard hours of complaints, and you can either shut up or get out. Perception isn’t reality everywhere; it is only the mantra in sick organizations totally removed from reality.

I saw a tweet the other day about the eight things that keep talented employees:

Talented employees stay when they're: 1 paid well 2 mentored 3 challenged 4 inspired 5 empowered 6 appreciated 7 on a mission 8 having fun

— Vala Afshar (@ValaAfshar) August 17, 2014

I’m normally not a fan of reducing human behavior to a list like this, but it seems pretty complete, and the words resonated.

As a technical leader I am a fan of metrics and dashboards: tests passing, code coverage, static analysis and metrics, velocity, bugs, takt time, and other indicators that you wouldn’t focus on individually, but are very useful collectively.

I wonder if, as a manager, assuming trust is in place, is it worthwhile to go over these things explicitly? To make a “private dashboard” to cover in 1-on-1s, and see where the problems are?

• If everyone is not feeling challenged, why is that? Is it because the work is boring? If so, why? Is it temporary grunt work that can be augmented with some side projects? Or is the product becoming less exciting to work on?
• Who doesn’t feel like they’re on a mission? Is it because they are disillusioned, or is the mission bullshit. Are more people disillusioned each month?
• Which individuals are trending up and down? Is the organization as a whole trending up or down? Which attributes are trending up and down? For all these questions, you must answer “why“!

I’m not sure this is something I will start using immediately, since I am just getting to know my team and I don’t want overly formal process to get in the way of a human connection. But it’s certainly something I would have done at CCP. It’s very convenient for bad managers to rationalize poor ratings, but perhaps some tracking on these eight points can be a start towards providing quantitative evidence of employee satisfaction.

Mike Cohn over at Mountain Goat Software says this in My Primary Criticism of Scrum:

In today’s version of Scrum, many teams have become overly obsessed with being able to say they finished everything they thought they would. This leads those teams to start with the safe approach. Many teams never try any wild ideas that could lead to truly innovative solutions.

I believe a great deal of this is the result of the move to shorter sprints, with two weeks being the norm these days. Shorter sprints leave less time to recover if a promising but risky initial approach fails to pan out.

Most teams I have joined have been working in two week sprints, and it has always seemed like the wrong length. In fact, at CCP, I championed “anything but two weeks.” I saw teams unable to get features done and tested in two weeks, which caused teams to not take the “shippable increment” idea very seriously. Nearly every sprint was a failed sprint in one way or another.

My preferred solution was to move to one week sprints. I’ve written about one week sprints before. If Agile is all about feedback, then halving your sprint length gives you double the feedback and chances to improve. You can try something out for a week, find it doesn’t work, and move on without putting a release at risk. A sprint twice as long carries twice the risk. I feel this, more than longer sprints, much better addresses Mike’s goal to get back to creativity. That said, it requires an intense commitment from the team to adopt Agile and eXtreme Programming development practices, such as through automated testing, shared code ownership, and more. It also requires a team that can function and perform well, and someone to coach. Without these things, one week sprints will fail.

My backup solution, when I feel a team isn’t yet ready to go through the transformation one-week sprints bring about, is to move to longer sprints. This does not encourage the behavior I ultimately want to see, but at least it allows more experimentation. More importantly, teams can take the “shippable increment” ideal seriously.

This reasoning can be taken to absurdity, but avoid it. I once had someone seriously suggest two releases a year was too often, and we should go down to one release a year, so we could make sure it was high quality and bug free. These people just need a reminder of the history of software (and often their own company). Even stuff like printer firmware can be iterated on less than 4 weeks.

In the end, I still prefer one week sprints, which isn’t surprising as I’m an XP advocate. But if a team isn’t able or willing to move to one week sprints, don’t default to two weeks.

In a previous article, commenter Robert Kist asked:

How are you going to judge if people trust you – what would your indicators be, if you decide to treat it as a metric?

This is tricky. I don’t think there’s a good answer. Bad managers, who are by definition less trusted, can easily rationalize away any attempt at quantification. One quantifiable measure would be employee turnover. However turnover tends to be ignored in the aggregate, since the sample size is necessarily small. Individual departures are blamed on “incompatibility” or “burnout,” when it is without exception due to mismanagement.

Another potentially useful metric is the employee survey. This is an unequivocal disaster, because there are just endless ways to explain away poor ratings. I’ve heard bad ratings blamed on bad weather!

What about the venerable “360 review”, where your direct reports and peers give feedback? This would require some sort of survey, and we know how that goes.

In the end, bad managers will be able to rationalize away poor trust metrics. Unfortunately they are also universally unable to qualitatively identify a lack of trust. Good managers, on the other hand, intuitively know whether trust is getting better or worse. They should make sure there are many avenues of feedback to inform this intuition, but I’m not sure any are worth quantifying, if they are quantifiable in the first place.

Someone please tell me when I start to become a bad manager, please.