Archive of articles classified as' "rant"

Back home

Automated testing shows a respect for employees

1/02/2015

In the tech-artists.org G+ community page there was a comment on a thread about unit testing:

A key factor in TA tools is the speed at which we need to deliver them, and our audience is considerably smaller than, say, engine tools code. Therefor it becomes somewhat hard to justify the time spent on writing the unit tests, and then maintaining them as the tools change or are ported or updated to match new APIs.

In other words: Testing is great, but we don’t have time for it. Or the common alternative: Testing is great, but it’s not feasible to test what we’re doing.

Codebases without tests manifest themselves in teams that are stressed and overworked due to an ever-increasing workload and firefighting. Velocity goes down over time. Meanwhile, I’ve never known a team with thorough test coverage that delivered slower than a team without automated tests. In fact I’ve observed teams that had no tests and crunched constantly, added tests and became predictable and successful, then removed the tests after idiotic leadership decisions to artificially increase velocity, and watched their velocity drop way down once the testing infrastructure, and especially culture, fell into disrepair.

Companies that do not require automated tests do not respect their employees, and do not care about the long-term health of the company. It’s that simple (or they are incompetent, which is equally likely). We know that no testing results in stress, overwork, and reduced quality. We know that more testing results in more predictability, higher quality, and happier teams. I would love to blame management, but I see this nonchalant attitude about testing just as often among developers.

The “do it fast without tests, or do it slow with tests” attitude is not just wrong, but poisonous. You are going to be the one dealing with your technical debt. You are the bottleneck on call because your stuff breaks. You are the one who doesn’t get to work on new stuff because you spend all your time maintaining your old crap. You are the one who is crunching to tread water on velocity.

I have a simple rule: I will not work at a job that doesn’t have automated testing (or would be in any way inhibited instituting it as the first order of business).

  • I have this rule because I love myself and my family. There are enough unavoidable opportunities to interrupt evenings and weekends for work reasons. It is irresponsible to add more ways for things to break.
  • I have this rule because I care for the people I work with. I want them to have the same option for work-life balance, and work with me for a long time.
  • I have this rule because I want the company I’ve decided to invest in (employment is the most profound investment!) to be successful in the long term. Not until the end of the quarter, or even until I leave, but for a long, long time.
3 Comments

How do you estimate that which you’ve never done?

29/01/2015

Have you heard about #noestimates? No? Well I’m sure you can guess what it is anyway. But reading the debates reminded me of a story.

While at Game Developer’s Conference a few years ago, I was arguing about estimation with a certain project manager, who, despite having no actual development experience, was in charge of development (Icelandic society is notoriously nepotistic).

“So, maybe no estimation works for your small projects, but when you have to do big projects, and you need to ask for budget, and coordinate many departments and offices, and you need to plan all this in advance, what do you do? How would you plan Incarna?”

Incarna was CCP’s expansion that introduced avatar/character-based “gameplay” into EVE Online. What shipped was the ability for your avatar to walk (not run!) around a room. It was massively over budget, behind schedule, and under delivered. A few months later, 20% of the company was laid off. There’s been no active development on Incarna since 2011, and World of Darkness- which continued to use Incarna’s core technology- was cancelled and the team laid off earlier this year. It was, quite simply, the biggest disaster I’ve seen or heard of my career.*

A character-based game is also something CCP had never done before. They are massively- MASSIVELY- more technologically complex than the “marbles in viscous fluid” EVE flight simulator. CCP did not have the in house experience, especially in Iceland, where most of the (very smart) engineering team had never worked on character based games.

So it was pretty hi-larious that a project manager was using Incarna as an example of why estimation is necessary. But cognitive dissonance is nothing new. Anyway, my response was:

“You don’t plan Incarna. You greenlight a month of development. At the end of a month, you see where things are. Do you keep going for another month? If you are happy with the spend and progress, keep going. If not, pull the plug. Once you can make a prediction at the start of a month, and it holds true for that month, and you do this two times in a row, maybe make a prediction for two months and see how it plays out.
You may pass a year this way. Well, a year isn’t a long time for developing a character-based MMO and game engine from scratch. But at the end of the year, you at least have some experience. But you keep going. If your velocity is consistently predictable, you estimate further out. Eventually, if you can get your velocity stable at the same time you’re growing and developing, you have a fighting chance.**
When your velocity isn’t stable, you reign things in and figure out. If you go through a year of missed month-long predictions, you need to change things drastically (or reboot entirely) if you hope to get something predictable.”

Nothing really insightful there of course- I’m just parroting what has worked me me and many, many others, from Lean-inspired methodologies (and this one in particular says traditional yearly budget cycles are responsible for many terrible business decisions).

A couple months ago I was asked if a significant new feature could get done by June. It would build on several months of foundation and other features. I responded that I was pretty confident that if we aim for June we would have it by September. My rationale, simply, was that previous similar projects shipped 3 or more months late, and I didn’t have enough experience with the team to give a more accurate estimate.

The best predictor of future behavior is past behavior. You need to create historical data before you can extrapolate and plan.

The historical data also needs to be “meaningful.” That is a much more nuanced topic, though.


* It should go without saying that disasters the scale of Incarna are 100% at the hands of management.

** On Star Wars: The Old Republic, management took an interesting strategy of driving velocity into the ground so that while it was terrible individually, it was at least stable. They could then increase the number of people resources and predict, pretty reliably, when it could ship. The game ended up costing about $200 million (I suspect much more, actually), but it wouldn’t have shipped otherwise.

1 Comment

Change should be the ally of quality

26/01/2015

In The Beauty of Testing, Steven Sinofsky writes:

…great testers understand one the cardinal rules of software engineering—- change is the enemy of quality.

This is not a cardinal rule. This is a outdated and obsolete mode of thinking. Change is how you discover great UX. Change is how you refactor and reduce technical debt. Change is how you incrementally improve both your product and code quality.

Maybe that’s too obvious, and clearly Sinofsky isn’t arguing for static software. More nuanced (and the rest of the piece provides that nuance) would be “change inevitably introduces bugs, and bugs reduce quality.”

This too I take issue with. Your codebase should be verifiably better after you fix a bug: you’ve found a shortcoming in your automated tests, so you add a test, and maybe refactor some stuff as well. Or, you’ve identified a bad experience, and can change it to be better in a controlled manner. A bug is an opportunity for improvement. Without bugs, it can be very difficult to improve.*

It can be difficult for anyone who hasn’t worked in a codebase with extensive testing to understand this. In most cases, fixing bugs is playing whack-a-mole. Whack-a-mole is unacceptable to me. Every change we make at Cozy is making the code clearer, simpler, better tested. It’s making the product smoother, faster, and more intuitive.

Change is necessary; it is up to you to determine if it is a friend or foe.


If you’re practicing disciplined development and automated testing and not creating many bugs, good job! This post isn’t for you :)

3 Comments

Undefining “technical debt”

6/01/2015

For me, technical debt is defined pretty loosely as stuff you don’t like in the code and need to change to keep up velocity. However, I’ve seen lots of articles lately discussing a precise definition of “technical debt.” I would sum them up as:

  • Technical debt is incurred intentionally. Sloppy code or bad architecture is not debt.
  • It is a business decision to incur technical debt.
  • It is a business decision to pay down technical debt.

I hate this characterization of technical debt. I hate it because it’s damaging. It assumes a conversation like this happens:

Manager: “How long to do this feature?”
Programmer: “We can do that feature in 4 weeks properly, or 2 weeks if we take shortcuts that will hurt our velocity in the future.”
Manager: “OK, take a shortcut and get it down ASAP.”
… 2 weeks later …
Manager: “How long to do this feature?”
Programmer: “We must spend 2 weeks paying down our technical debt, then another 2 weeks to do the feature.”
Manager: “That sounds fine.”

Every muscle in my body twinges when I think about this. Quality is not something you can put off to later. The idea that a team would do a sloppy job but have the rigor to repay it later is unbelievable. The closest I’ve seen is rewriting a system after years of shortcuts, which often does not end well. This mentality goes along with “how many bugs you have should be a business decision”. This isn’t OK. Do not write something you do not plan on living with. Do not place the responsibility of doing a good job on the business. I find it sad that a programmer would think such behavior acceptable. This is your life. This is your code. Take some responsibility. Take pride in your work.

Or don’t, and sling garbage while getting paid a pretty penny. Just don’t pretend you’re respecting your craft.


(I just want to take a moment to give credit to the team at Cozy. We recently had a couple weeks of crunch. The team delivered fully tested code the entire time).

9 Comments

Holiday (product) shipping

2/01/2015

This was an interesting holiday season, work-wise, for three reasons.

First: My work was closed down from Dec 20th to Jan 4th (except for Customer Support and whichever developer was on firefighting duty, though that is all remote). We shipped two large products on December 17th, which was a bit too close for comfort, but things went OK and it gave us a few days to fix issues.

Second: I was working a couple hours a day while my son napped. I have quite a backlog of pull requests waiting to get in.

Third: On December 31st at about 5pm, we realized our emails hadn’t been going out. Our email service decided to ship 43,000 lines of code the day before, which resulted in a partial outage for some customers (they sent us success responses but things then broke internally).

What lessons did I learn?

First, if you’re going to ship two days before vacation, make sure your work is solid. We had one deployment on Sunday the 21st for some bugs we didn’t want to live with for 2 weeks, but other than that no new work has gone out. We shipped some solid code, thankfully.

Second, if you’re going to work over a holiday, don’t generate work for others. I really want to get the work I’ve been doing out to production, which would require 1) a code review and 2) a deploy of new code. Even if I skipped code review and deployed myself, if shit hit the fan or I introduced some new bug, I’m making work for others. I took a lot of discipline but I’m proud to say that I have fifteen open pull requests and not a single one is reviewed yet. It’ll be a busy Monday and Tuesday but that’s better than messing with peoples’ vacation.

Third, two weeks is a really long time to shut down. In some ways, shutting down is great, as I’ve written about before. But it sucks not having a good way to get fixes and improvements out to customers. There are a lot of considerations here. I’m not sure what we’ll do next year. It’ll largely be up to the team.

Fourth, you should never, ever ship something directly before a holiday or before you go on vacation. It’s immature and unacceptable. You not only screw over your team when something goes wrong, you screw over everyone depending on your product. They need to jump into action and figure out what’s going on, how to mitigate things, respond to customer complaints, etc. I cannot believe I need to tell anyone this. Don’t ship directly before a holiday.

Anyway, just some thoughts. Happy New Year!

No Comments

We’re not so different, you and I

21/12/2014

Ben Sandofsky wrote a post about why QA departments are still necessary, specifically with regards to mobile app development. He makes a good point: mobile apps create a distribution bottleneck that makes very rapid iteration impossible. I agree, and this is a good angle to think about. I would have been happy with an article focused on this.

Ben is clearly a talented guy but this post was insane. In a literal sense. It is a rant for anti-Agile curmudgeons at best, and would leave me questioning the experiences of anyone that thinks this way at worst.

Websites ship embarrassing bugs all the time. They get away with it because they didn’t ship it to all users. You roll-out everything out to 1% of users, and watch your graphs. If things look good, slowly roll out to 100%.

The idea that this is this sort of incremental rollout is ubiquitous amongst web developers is crazy. It requires infrastructure, code designed to support split testing, experienced operations engineers, robust monitoring, a disciplined process, and more. The institutions with this sort of sophistication all have strong automated testing environments. Which brings me to my next issue:

I think automated testing accelerates development, but I haven’t seen a direct correlation between testing and quality. On projects with massive, high quality test coverage, I’ve seen just as many bugs slip through as projects with zero coverage.

This is the software equivalent to climate change denial. Where does this experience come from? I am not sure I’d be able to find a single developer who would corroborate this. Oh, right:

Tell a game developer you don’t need [QA], they’ll tell you you’re nuts.

The game industry is full of these folks who believe what they are doing is such an untestable snowflake. Unsurprisingly, games have historically been the buggiest software around. Never, ever look at game development as an example of how to do QA right. Not just automated testing, but manual QA too.

…a great QA team is far from a bunch of monkeys clicking buttons all day.

Game development has a hallmark technique of hiring masses of QA people and have massive layoffs at the end of projects. There is an entire website dedicated to tales of horror from QA people. It makes The Daily WTF look like paradise.

Take the unicorn of “two week release cycles.” As you build infrastructure for faster releases, simple code becomes unwieldy. Tasks that should take hours take weeks.

What does this even mean? There are endless apps on two week release cycles. I am confused how building infrastructure for faster iterations ends up adding complexity to simple code or tasks.

Disciplined development is a lost art.

You could make this argument when we moved away from punch cards. But the idea that success in mobile apps is achieved through discipline, but success on the web can be achieved by recklessness, is beyond baseless. It’s downright insulting.

I consider it a tragedy that, when faced with the reality of App Store distribution bottlenecks, Ben’s answer is to go back to the process of yesteryear and throw out the lessons we’ve learned. Why not invent new ways of building in quality? New ways of iterating on apps faster? There are so many interesting problems to solve.

Finally, Ben cautions:

Today, any web developer who wants to stay employed has learned to build apps. If web companies want to remain relevant, they’ll have to do the same.

I have a better warning. Don’t throw away the incredible advances we’ve made over the last decade. Don’t downplay the success and rate of innovation in web development as something that doesn’t apply. Don’t throw away the universal “good idea-edness” of automated testing. Don’t rely on a separate department to enforce quality. Don’t stop looking for ways to make development better.

1 Comment

Long live Slack, down with egotistical email

17/11/2014

We use Slack for team communication at Cozy. I struggled with the transition. When I reflected on my struggles, it made me better understand what a destructive format email is for workplace communication.

A quick disclaimer. This is only about work communication and not personal communication. I love email. I think email will be around for a long time and I will lament if and when it goes away. I just don’t think we should be using email for work.

Oration is the highest form of feeding an ego. You craft your message carefully. You research, write, and rehearse. Finally, you take the stage. You command everyone’s attention. And once you’re done, an important topic has been thoroughly addressed and everyone can go on with their lives, better off after hearing what you said.

Email is oratory without the speaking* (or skill). My problems with email stem from when it is used for one-way communication. I suspect that most emails I’ve ever received from anyone in management have been one-way. Generally these emails are meant to, first and foremost, communicate the sender/manager’s self-importance. Often the email contains a nugget of actual information which should be hosted elsewhere. Sometimes the email is an announcement no one understands. And as a rule, you can’t rely on people reading the email you send anyway.

When you craft a long email, like an orator crafts a speech, it is an ego boost. Each one is a masterpiece. You are proud of your fine writing. When you craft a long chat message, on the other hand, you look like a dramatic asshole. It puts in stark perspective how awful the written format is for important or high-bandwidth communication. I’ve never seen someone post a 300-word message to chat. How many 300-word emails do you have in your inbox?

Removing email also levels the playing field for communication. You don’t need to be a manager or orator. Everything you write has a visibility you can’t change. You choose your audience based on topic. Is there a question about a product’s design? Well, it goes into the product or design channel, whether you are Executive Emperor or QA Associate II. Also, no one really wants to read your dramatic flair so please keep it short and to the point.

I used to get frustrated when I’d write an excellent email, send it out, and within a few minutes someone would reply with a message like “Yeah, just to build on what Rob said, it’d be a good idea to do X.” You idiot! You are an Ice Cream Truck driving through the State of the Union. But of course, the problem was mine, playing a manipulative game, focusing too much on this amazing message I’d created. Sometimes these emails would be about the manipulative games people were playing and how we weren’t focused on the employees and customers and things that were actually important.

Email in the workplace is a systematic problem. We take it for granted. We use it constantly. We don’t question it. But email has a cost. It feeds into the already inflated ego of managers. It encourages one-way communication. It is wonderful for grandstanding. We spend a lot of time crafting museum-quality correspondence no one wants to read. And in the end, there are better ways to accomplish what we use it for.


* One of the greatest “speeches” of all time, Pro Milone by Cicero, was written, not spoken. We know great orators by their writing, not their speaking.

No Comments

First, do no harm

13/11/2014

From a wonderful post by Matt Williams about the type of business he is looking for:

A Business Manifesto
We are uncovering better ways of running a business and helping others do it.
Through this work we have come to value:
People and interactions over profits and prestige
Quality service over quantity of service
Customer relationships over contract negotiation
Flexibility over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

In a nutshell I want to work for a company which values people — both inside and out of the company. I want to work where people strive to do things right.

When I go home, I want to be able to look in the face of my daughter and not have to make excuses for the work that I do and the effect it has on others.

Sums things up nicely (and definitely what we aspire to at Cozy, by the way we’re hiring).

It is a reason I left the video games industry. I wanted to use my skills to do something I felt was more constructive.

But more than that, I was amazed and frustrated with how the industry was run (almost as bad as films). Mass layoffs even on successful projects. Over-managed projects that go on for 4, 5, 6 years and are cancelled. Creating an exploitative product in order to milk a customer base. Huge budgets, huge marketing, appeals to lowest common denominators (often sexual). There are good companies but the business models are so insane that you can be around for 10 years and fold tomorrow.

2 Comments

If you hear “perception is reality” you’re probably being screwed

27/10/2014

I was once told in a performance review that “perception is reality.” I was infuriated, and the words stuck in my mind as the most toxic thing a manager could say to an employee. I have avoided writing about it, but the “This American Life” episode about Carmen Segarra’s recordings at the Fed has inspired me to change my mind. Here’s the relevant section, emphasis mine:

Jake Bernstein: Carmen says this wasn’t an isolated incident. In December– not even two months into her job– a business line specialist came to Carmen and told her that her minutes from a key meeting with Goldman executives were wrong, that people didn’t say some of the things Carmen noted in the minutes. The business line specialists wanted her to change them. Carmen didn’t.

That same day, Carmen was called into the office of a guy named Mike Silva. Silva had worked at the Fed for almost 20 years. He was now the senior Fed official stationed inside Goldman. What Mike Silva said to Carmen made her very uncomfortable. She scribbled notes as he talked to her.

Carmen Segarra: I mean, even looking at my own meeting minutes, I see that the handwriting is like nervous handwriting. It’s like you can tell. He started off by talking about he wanted to give me some mentoring feedback. And then he started talking about the importance of credibility. And he said, you know, credibility at the Fed is about subtleties and about perceptions as opposed to reality.

Well shit, if that doesn’t sound familiar. Here I was, doing work that was by all measures extremely successful, yet pulled into a feedback meeting to be told “perception is reality.”

Let me tell you what “perception is reality” means, and why you should plan on leaving your job the moment you hear it:

The arbitrary opinions of your manager’s manager defines your situation. And they don’t like what you’re doing.

Your manager may be well-meaning (mine was, as was Mike Silva), but at the point you get this “perception is reality” bullshit, you can be sure there’s nothing that they are going to do to help you. Someone above them has taken notice, your manager has probably heard hours of complaints, and you can either shut up or get out. Perception isn’t reality everywhere; it is only the mantra in sick organizations totally removed from reality.

1 Comment

Escaping the Windows prison

8/09/2014

My friend Brad Clark over at Rigging Dojo is organizing a course on Maya’s C++ API (I am assuming it is Maya but could be another program). He had a question about student access to Visual Studio, to which I responded:

As a programmer, the experience on Windows is pretty horrific. No built-in package manager. A shell scripting language designed in hell. A command line experience that is beyond frustrating. A tradition of closed-source and GUI-heavy tools that are difficult or impossible to automate. A dependence on the registry that still confounds me.

My eyes weren’t opened to this reality until I switched to Linux. I was on a Windows netbook that was barely working anymore. I installed Linux Mint XFCE and suddenly my machine was twice as fast. But the much more important thing that happened was exposure to modes of developing software that I didn’t even know existed (Python, Ruby, and Go made a lot more sense!). Programming in Windows felt like programming in prison. Developing on Linux made me a better programmer. Furthermore, If I didn’t end up learning the Linux tools and mindset on my own, working in the startup world would be basically impossible.

Programming on Windows revolves around Visual Studio and GUI tools. If you need evidence at how backwards Windows development is, look no further than Nuget. This was a revolution when it was released in 2010, changing the way .NET and Windows development was done. In reality, the release of Nuget was like electricity arriving in a remote village. It took ten years for C# to get a package manager. And it is for the rich villagers only: older versions of Visual Studio don’t work with Nuget.

The ultimate problems Windows creates are psychological. The technical differences change the way you think. It echoes the “your Python looks like Java” problem. Is the Windows mentality what we want students to take on? My last two jobs have been on Linux/OSX. I honestly cannot imagine having kept my head above water if I didn’t have a few years of self-taught Linux experience.

Windows still dominates in desktop OS popularity. That is all the more reason to make sure we are exposing student programmers to alternative ways of developing. I want new developers to be exposed to things outside the Microsoft ecosystem. It is too easy to become trapped in the Windows prison.

5 Comments