Archive of articles classified as' "testing"

Back home

Automated testing shows a respect for employees

1/02/2015

In the tech-artists.org G+ community page there was a comment on a thread about unit testing:

A key factor in TA tools is the speed at which we need to deliver them, and our audience is considerably smaller than, say, engine tools code. Therefor it becomes somewhat hard to justify the time spent on writing the unit tests, and then maintaining them as the tools change or are ported or updated to match new APIs.

In other words: Testing is great, but we don’t have time for it. Or the common alternative: Testing is great, but it’s not feasible to test what we’re doing.

Codebases without tests manifest themselves in teams that are stressed and overworked due to an ever-increasing workload and firefighting. Velocity goes down over time. Meanwhile, I’ve never known a team with thorough test coverage that delivered slower than a team without automated tests. In fact I’ve observed teams that had no tests and crunched constantly, added tests and became predictable and successful, then removed the tests after idiotic leadership decisions to artificially increase velocity, and watched their velocity drop way down once the testing infrastructure, and especially culture, fell into disrepair.

Companies that do not require automated tests do not respect their employees, and do not care about the long-term health of the company. It’s that simple (or they are incompetent, which is equally likely). We know that no testing results in stress, overwork, and reduced quality. We know that more testing results in more predictability, higher quality, and happier teams. I would love to blame management, but I see this nonchalant attitude about testing just as often among developers.

The “do it fast without tests, or do it slow with tests” attitude is not just wrong, but poisonous. You are going to be the one dealing with your technical debt. You are the bottleneck on call because your stuff breaks. You are the one who doesn’t get to work on new stuff because you spend all your time maintaining your old crap. You are the one who is crunching to tread water on velocity.

I have a simple rule: I will not work at a job that doesn’t have automated testing (or would be in any way inhibited instituting it as the first order of business).

  • I have this rule because I love myself and my family. There are enough unavoidable opportunities to interrupt evenings and weekends for work reasons. It is irresponsible to add more ways for things to break.
  • I have this rule because I care for the people I work with. I want them to have the same option for work-life balance, and work with me for a long time.
  • I have this rule because I want the company I’ve decided to invest in (employment is the most profound investment!) to be successful in the long term. Not until the end of the quarter, or even until I leave, but for a long, long time.
4 Comments

Change should be the ally of quality

26/01/2015

In The Beauty of Testing, Steven Sinofsky writes:

…great testers understand one the cardinal rules of software engineering—- change is the enemy of quality.

This is not a cardinal rule. This is a outdated and obsolete mode of thinking. Change is how you discover great UX. Change is how you refactor and reduce technical debt. Change is how you incrementally improve both your product and code quality.

Maybe that’s too obvious, and clearly Sinofsky isn’t arguing for static software. More nuanced (and the rest of the piece provides that nuance) would be “change inevitably introduces bugs, and bugs reduce quality.”

This too I take issue with. Your codebase should be verifiably better after you fix a bug: you’ve found a shortcoming in your automated tests, so you add a test, and maybe refactor some stuff as well. Or, you’ve identified a bad experience, and can change it to be better in a controlled manner. A bug is an opportunity for improvement. Without bugs, it can be very difficult to improve.*

It can be difficult for anyone who hasn’t worked in a codebase with extensive testing to understand this. In most cases, fixing bugs is playing whack-a-mole. Whack-a-mole is unacceptable to me. Every change we make at Cozy is making the code clearer, simpler, better tested. It’s making the product smoother, faster, and more intuitive.

Change is necessary; it is up to you to determine if it is a friend or foe.


If you’re practicing disciplined development and automated testing and not creating many bugs, good job! This post isn’t for you :)

4 Comments

Technical debt metaphors get it so wrong

21/01/2015

In my previous post about technical debt, I explained how modern definitions of technical debt are harmful. Now I turn my attention to equally harmful metaphors.

Viktoras Makauskas made the following metaphor in a comment on my last post. This is a pretty perfect stand-in for metaphors I’ve read in other articles that harmfully define technical debt.

Imagine your car gets a strange rattle. You go to your mechanic and he says, “it’s your exhaust pipe holder, you need to replace it, but it’s gonna take a while to order a part and ship it, so just park your car here and come back in a week”. You say “no, I have this weekend trip planned, is there something we can do now?”. They say “yeah, we’ll put a strap on it meanwhile, just drive a little more careful and it should hold, but make sure to come back and do a proper fix”. Mechanic charges you now, and then a bit later.

This seems sensible on first read. But upon closer inspection, it’s quite clear the roles here are totally wrong*:

  • The mechanic is the programmer (the role of the “expert”). Well, a mechanic may or may not see your car ever again. They do not have a vested interest in your choice. A mechanic’s relationship to a car is totally different from a programmer’s relationship to code.
  • “You” are the “business” (the role of the “stakeholder”). The metaphor assumes that if you are irresponsible, it only impacts you (it’s your car, your money, your time). This is a problem. A programmer is impacted by business decisions in a way a mechanic is not impacted by whether you fix your car now or later.

This isn’t a simple language problem. It is a fundamental misunderstanding of roles that is naive to the way software development works. Programmers will be the primary sufferers of technical debt. Eventually the business will suffer with a slower pace of innovation and development and higher turnover. But well before that, programmers will be fixing (and refixing) obscure bugs, will bristle under management that tells them to go faster, will be working extra hours to try to improve things, and will eventually burn out. The business will only suffer once real damage has been done to a programming team, and many have given up.

This is why control of technical debt must be in the hands of programmers. Definitions or metaphors that urge otherwise are actively harmful.

Let me close by pointing out I’m just repeating what Ward Cunningham has already written about the original technical debt metaphor. The article ends with:

A lot of bloggers at least have explained the debt metaphor and confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later and thinking that that was the primary source of debt.
I’m never in favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of a problem even if that understanding is partial.

Thanks Ward.


* There are also a couple other problems with this metaphor. First, if “you” and the mechanic are the same person, and responsible for both business and implementation? In that case, there’s no need for a metaphor at all. Second, what happens if the exhaust fails? Do you become stranded? Does the car catch fire? What’s presented here is a false choice between a “correct” solution (replacement) or a “sloppy” solution (strapping it on). Why not rent a car? If there’s no responsible-but-relatively-cheap decision (there almost always is!), it’s still never acceptable to make an irresponsible decision.

4 Comments

Undefining “technical debt”

6/01/2015

For me, technical debt is defined pretty loosely as stuff you don’t like in the code and need to change to keep up velocity. However, I’ve seen lots of articles lately discussing a precise definition of “technical debt.” I would sum them up as:

  • Technical debt is incurred intentionally. Sloppy code or bad architecture is not debt.
  • It is a business decision to incur technical debt.
  • It is a business decision to pay down technical debt.

I hate this characterization of technical debt. I hate it because it’s damaging. It assumes a conversation like this happens:

Manager: “How long to do this feature?”
Programmer: “We can do that feature in 4 weeks properly, or 2 weeks if we take shortcuts that will hurt our velocity in the future.”
Manager: “OK, take a shortcut and get it down ASAP.”
… 2 weeks later …
Manager: “How long to do this feature?”
Programmer: “We must spend 2 weeks paying down our technical debt, then another 2 weeks to do the feature.”
Manager: “That sounds fine.”

Every muscle in my body twinges when I think about this. Quality is not something you can put off to later. The idea that a team would do a sloppy job but have the rigor to repay it later is unbelievable. The closest I’ve seen is rewriting a system after years of shortcuts, which often does not end well. This mentality goes along with “how many bugs you have should be a business decision”. This isn’t OK. Do not write something you do not plan on living with. Do not place the responsibility of doing a good job on the business. I find it sad that a programmer would think such behavior acceptable. This is your life. This is your code. Take some responsibility. Take pride in your work.

Or don’t, and sling garbage while getting paid a pretty penny. Just don’t pretend you’re respecting your craft.


(I just want to take a moment to give credit to the team at Cozy. We recently had a couple weeks of crunch. The team delivered fully tested code the entire time).

10 Comments

We’re not so different, you and I

21/12/2014

Ben Sandofsky wrote a post about why QA departments are still necessary, specifically with regards to mobile app development. He makes a good point: mobile apps create a distribution bottleneck that makes very rapid iteration impossible. I agree, and this is a good angle to think about. I would have been happy with an article focused on this.

Ben is clearly a talented guy but this post was insane. In a literal sense. It is a rant for anti-Agile curmudgeons at best, and would leave me questioning the experiences of anyone that thinks this way at worst.

Websites ship embarrassing bugs all the time. They get away with it because they didn’t ship it to all users. You roll-out everything out to 1% of users, and watch your graphs. If things look good, slowly roll out to 100%.

The idea that this is this sort of incremental rollout is ubiquitous amongst web developers is crazy. It requires infrastructure, code designed to support split testing, experienced operations engineers, robust monitoring, a disciplined process, and more. The institutions with this sort of sophistication all have strong automated testing environments. Which brings me to my next issue:

I think automated testing accelerates development, but I haven’t seen a direct correlation between testing and quality. On projects with massive, high quality test coverage, I’ve seen just as many bugs slip through as projects with zero coverage.

This is the software equivalent to climate change denial. Where does this experience come from? I am not sure I’d be able to find a single developer who would corroborate this. Oh, right:

Tell a game developer you don’t need [QA], they’ll tell you you’re nuts.

The game industry is full of these folks who believe what they are doing is such an untestable snowflake. Unsurprisingly, games have historically been the buggiest software around. Never, ever look at game development as an example of how to do QA right. Not just automated testing, but manual QA too.

…a great QA team is far from a bunch of monkeys clicking buttons all day.

Game development has a hallmark technique of hiring masses of QA people and have massive layoffs at the end of projects. There is an entire website dedicated to tales of horror from QA people. It makes The Daily WTF look like paradise.

Take the unicorn of “two week release cycles.” As you build infrastructure for faster releases, simple code becomes unwieldy. Tasks that should take hours take weeks.

What does this even mean? There are endless apps on two week release cycles. I am confused how building infrastructure for faster iterations ends up adding complexity to simple code or tasks.

Disciplined development is a lost art.

You could make this argument when we moved away from punch cards. But the idea that success in mobile apps is achieved through discipline, but success on the web can be achieved by recklessness, is beyond baseless. It’s downright insulting.

I consider it a tragedy that, when faced with the reality of App Store distribution bottlenecks, Ben’s answer is to go back to the process of yesteryear and throw out the lessons we’ve learned. Why not invent new ways of building in quality? New ways of iterating on apps faster? There are so many interesting problems to solve.

Finally, Ben cautions:

Today, any web developer who wants to stay employed has learned to build apps. If web companies want to remain relevant, they’ll have to do the same.

I have a better warning. Don’t throw away the incredible advances we’ve made over the last decade. Don’t downplay the success and rate of innovation in web development as something that doesn’t apply. Don’t throw away the universal “good idea-edness” of automated testing. Don’t rely on a separate department to enforce quality. Don’t stop looking for ways to make development better.

1 Comment

Grabbing for good enough

15/12/2014

Uncle Bob, who I consider my favorite programming writer, had a post a few weeks ago titled “Thorns around the Gold“. In it he describes how writing tests for your core functionality first can be harmful. Instead, Uncle Bob prefers to probe for “thorns” around the “gold” first.

I shy away from any tests that are close to the core functionality until I have completely surrounded the problem with passing tests that describe everything but the core functionality. Then, and only then, do I go get The Gold.

I haven’t been doing TDD for nearly as long as Uncle Bob but I was shocked to read this. I’ve always learned and taught that you should create positive tests first, and only need as many negative tests as you feel are warranted. While you may not grab the gold immediately, you at least step towards the gold. How many thorns you expose is a judgement call. In Python, most people don’t even bother validating for None inputs, and instead just let things raise (or not). Of course, this depends on your users. For libraries limited to one internal application, I wouldn’t “probe many hedges.” For open source libraries, I validate pretty aggressively.

Of particular interest was this:

I often find that if all the ancillary behaviors are in place before I approach the core functionality, then the core functionality is much easier to implement.

I always thought you should only program what you need and no more. It seems very strange to assume the ancillary behaviors will be needed. It seems like a violation of YAGNI.

I have been trying to reconcile Uncle Bob’s advice here, and the TDD best practices I’ve learned and developed. But I cannot. Either I’ve been receiving and giving years of bad advice, or Uncle Bob has made a rare mistake.

2 Comments

The QA Department Mindset

8/12/2014

From this post by Rands, titled “The QA Mindset”:

At the current gig, there’s no QA department. […]

My concern is that the absence of QA is the absence of a champion for aspects of software development that everyone agrees are important, but often no one is willing to own. Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective. Traditionally, they are known as the folks who break things, who find bugs, but QA’s role is far more important. It’s not that QA can discover what is wrong, they intimately understand what is right and they unfailingly strive to push the product in that direction.

I believe these are humans you want in the building.

At my current job, we don’t have a QA department either. And like Rands, I wasn’t comfortable at first. I’ve worked on teams without QA, but an entire company without a QA Department? I’ve certainly had questions about the use of a QA department, but does that mean they are a bad idea?

Yes, and this line in Rands’ defense is why:

Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective.

I am a staunch believer of “building quality in.” Every bug that slips out is a failure of your development process. The way to higher quality is not to find, or fix, more bugs. It’s to avoid them in the first place.

If you rely on QA to champion unit testing, automation, bug tracking, and quality metrics, your development process is lacking its most important tools and measures to improving quality. Quality can’t be imposed by QA, it must grow out of enabled and engaged development teams.

I have a saying: “Don’t hire to fix a problem.” If you have a quality problem, hiring a QA department isn’t going to fix it. You instead hide the systematic problems that cause quality issues in the first place.

This is not to say “the QA mindset” isn’t valuable. It is. One of my best hires was Bjorgvin Reynisson, who was a Test Engineer at Nokia and I hired as a QA Engineer at CCP. He was embedded with the graphics engine team and he helped them develop extensive automated correctness and performance testing systems. He worked with them to recognized holes in their process and test coverage. He helped with tracking issues and increasing quality. This is the “QA Mindset” I treasure, and this type of person is invaluable to development teams. Bjorgvin unlocked a latent “culture of quality” in the team he was a part of.

I contrast this “QA Mindset” with the “QA Department Mindset“. The QA Department Mindset has two damning characteristics. First, it is innately adversarial, as Rands notes.

Yes, there is often professional conflict between the teams. Yes, I often had to gather conflicting parties together and explain[…]

Second, it is by definition a separate department, which creates obstacles to better integrating engineering and QA.

Bjorgvin should be spending time with his teammates and the rest of the developers figuring out how to improve the entire development process. He should not be spending time with other QA personnel focused on QA functions. When I was Technical Director for EVE Online, I made sure there were minimal discussions gated by job title. Talk of a trade went on in Communities of Practice, which were open to all. Sometimes this didn’t happen, and those times were mistakes.

Like Rands says:

Yes, we actually have the same goal: rigorously confirming whether or not the product is great.

If that’s the case, QA should not be broken out into a separate department. QA should be working side by side, reporting into the same people, measured by the same success metrics, contributing to the holistic success of an entire product.

I love the QA Mindset. It’s tragic that having a QA Mindset gets confused with having a QA Department.

2 Comments

Behavioral testing is the bee’s knees

10/11/2014

I have been doing Test Driven Development (TDD) with xUnit-based frameworks (like unittest and NUnit) for a number of years now, and started with RSpec in Ruby and Jasmine in JavaScript a few months ago. RSpec and Jasmine are behavioral testing frameworks that facilitate Behavioral Driven Development (BDD). BDD is really no different from “normal” TDD except for the frameworks used. BDD frameworks facilitate a different way of designing tests.

Behavioral testing is awesome and makes xUnit-style testing seem barbaric and uncivilized in comparison. I didn’t see the big deal for a long time. My xUnit style tests served me just fine. But now I’m hooked.

If you are a TDD person (or do any unit testing), I encourage you to try out BDD. You should get the hang of it quickly. It can take a little while to learn BDD idioms, but once you start doing things like custom matchers, it’s absolutely amazing.

In future posts I’ll try to discuss more of the differences between TDD and BDD, but for now I just provide a hearty endorsement of BDD and frameworks like RSpec and Jasmine.

3 Comments

Two weeks is the worst sprint length

15/09/2014

Mike Cohn over at Mountain Goat Software says this in My Primary Criticism of Scrum:

In today’s version of Scrum, many teams have become overly obsessed with being able to say they finished everything they thought they would. This leads those teams to start with the safe approach. Many teams never try any wild ideas that could lead to truly innovative solutions.

I believe a great deal of this is the result of the move to shorter sprints, with two weeks being the norm these days. Shorter sprints leave less time to recover if a promising but risky initial approach fails to pan out.

Most teams I have joined have been working in two week sprints, and it has always seemed like the wrong length. In fact, at CCP, I championed “anything but two weeks.” I saw teams unable to get features done and tested in two weeks, which caused teams to not take the “shippable increment” idea very seriously. Nearly every sprint was a failed sprint in one way or another.

My preferred solution was to move to one week sprints. I’ve written about one week sprints before. If Agile is all about feedback, then halving your sprint length gives you double the feedback and chances to improve. You can try something out for a week, find it doesn’t work, and move on without putting a release at risk. A sprint twice as long carries twice the risk. I feel this, more than longer sprints, much better addresses Mike’s goal to get back to creativity. That said, it requires an intense commitment from the team to adopt Agile and eXtreme Programming development practices, such as through automated testing, shared code ownership, and more. It also requires a team that can function and perform well, and someone to coach. Without these things, one week sprints will fail.

My backup solution, when I feel a team isn’t yet ready to go through the transformation one-week sprints bring about, is to move to longer sprints. This does not encourage the behavior I ultimately want to see, but at least it allows more experimentation. More importantly, teams can take the “shippable increment” ideal seriously.

This reasoning can be taken to absurdity, but avoid it. I once had someone seriously suggest two releases a year was too often, and we should go down to one release a year, so we could make sure it was high quality and bug free. These people just need a reminder of the history of software (and often their own company). Even stuff like printer firmware can be iterated on less than 4 weeks.

In the end, I still prefer one week sprints, which isn’t surprising as I’m an XP advocate. But if a team isn’t able or willing to move to one week sprints, don’t default to two weeks.

1 Comment

A short letter to a unit testing newcomer

18/08/2014

One of my friends asked how to get started with unit testing and Test Driven Development and figured I could write a short post. I also mention TDD a few times in my book so I think it could use some more attention.

I got started with unit testing when I wrote a smallish Python project, and realized I couldn’t maintain large Python projects without tests*. So I wrote unit tests for my code. I then quickly got into TDD. I don’t remember what resources I used, but there are plenty.

I’d encourage you to start unit testing by writing your code first, then the tests. As soon as this becomes easy and is paying off, start writing the tests first (TDD). You will come to appreciate that this is a powerful way of thinking and working. Even if you don’t ultimately stick with this approach, it is worth becoming skilled with it. You will uncover many suboptimal areas in your coding process that are otherwise extremely difficult to find and fix.

Keep in mind that learning TDD isn’t like already knowing Mercurial, then reading a book about Git**, and then being skilled with Git because you are skilled with Mercurial. You are embarking on a long journey, and will need to refer to many tutorials, blogs, and books. You will do some TDD, and look back on that code and process in a year and be horrified, just like you were when you started programming. Do not think of unit testing and TDD like learning a new framework or library. Think of it like learning how to program all over again.

So I don’t know exactly where to get started, only that you must, and keep going once you do start.


* I’d soon realize that no project was really maintainable without tests.

** Pro Git is my favorite Git book, by the way.

1 Comment