On choosing software by “what is best for the business”

by Rob Galanakis on 3/03/2015

When people are discussing what language/framework/library to use for something, the general criteria people talk about is “what best solves the business problem.”

  • This criteria is used to justify rewriting backend services in Go, rather than sticking with Python. Or not.
  • It’s used explain why you wrote a new CRUD app in node, even though you’re already using Ruby. Or not.
  • It’s used to choose between frameworks like Watir or Capybara, even though they’re basically the same thing. Or not.
  • It’s used to introduce superior programming patterns into legacy codebases. Or not.
  • It’s used to introduce new unit test runners or libraries. Or not.

“Best solution for the business problem” is used to justify all manner of decisions that are risky and without worthwhile benefits. Likewise, it’s used to justify all manner of decisions that are restrictive and regressive.

I’m sorry to rat on my fellow developers but choosing by “what best solves the business problem” is a load of bullshit.

It’s much more honest to just admit that technology choices are made from a desire to work with a new technology, or because a technology is familiar.

  • “We have approval to rewrite this service. I’m tired of working in dynamic languages, and I’d like to try Go.”
  • “This is a small internal app, and I wanted to try node.”
  • “I am familiar with Capybara, and will be writing most of the tests, so I’d like to use that.”
  • “I am uncomfortable introducing a new programming pattern that I am unfamiliar with, even though it’s better.”
  • “I don’t like using this library, and the one I do like can live side-by-side, so I’d like to add it.”

I actually don’t have a problem with deciding this way. In fact, I think it’s a good thing! I want to keep employees happy. But it’s important to be clear about how you expect a codebase and culture to evolve (encourage or discourage change). I want to understand why and how we actually make decisions, so we can get better. Honest discussion leaves fewer loose ends, and less surface area for future criticism.

No Comments

What do you want next?

by Rob Galanakis on 24/02/2015

We are focusing on A and B, and in a month or so we’ll start focusing on C, while also keeping focus on A and B.

Sound familiar?

When we do prioritization at work, I insist we have a single column of priorities or coarse features. In other words, “what do you want delivered next?”*

If a team or person isn’t working on one of the top two or three priorities, they’re doing unimportant, and possibly counter-productive, work. You’d be amazed how many people are working on things someone arbitrarily said was important, which aren’t inline with actual priorities.

You’d be even more amazed how unimportant most “high priority” work is when it needs to be stacked along with everything else. A feature can easily sit at the number 4 spot for months. Just be careful work doesn’t move up the queue just because it’s been in the queue. I don’t think this is a problem, though, because when you tell a product person “we are only executing on the next 2 things to deliver” they are going to have to make hard decisions.

I’ve worked on projects from 10 to 500 people, and generally the times we were humming along was when we had one or two priorities. We ended up producing crap when we had n priorities (where n is often the number of people or teams). Big teams don’t mean more priorities. It is just the granularity of the priorities that changes.

This sort of rigid, columnar prioritization communicates to product people that work only gets done when it’s at the top of the column. I’ve run across countless people, both managers and developers, who just sort of, well, expect that stuff just sort of, well, gets done, somehow. And generally it appears as if things are getting done, until everyone finds out they weren’t really. Are there significant bugs in some old system? It’s not fixed until it’s a priority. Is that new system still unpolished? It’s not improved until it’s a priority. Want to build something that requires some serious infrastructure? Well, that infrastructure stays at the top of the column until it’s done, to the exclusion of other work. Do you want good tools? Well, it means you aren’t going to get features.

It’s an extremely simple and powerful technique, and I highly recommend it if you are having trouble coordinating a product group.


* This doesn’t include ongoing product support, small fixes, and improvements. I think you need a way to handle this outside of normal feature development teams, with some sort of “live support” that can react quickly. A topic for another post.

No Comments

Anxiety causes selfish behavior

by Rob Galanakis on 17/02/2015

BPS Research Digest is a great site, highly recommended for anyone interested in why people behave the way that they do. A little while ago, they reported on a study where anxious participants were more likely to cheat and excuse their own unethical behavior than the control group.

When we’re stressed out and feeling threatened, our priority becomes self-preservation. According to new research, this defensive mode even affects our morality, making us more likely to cheat and excuse our own unethical behaviour.

What’s striking is the cause of the anxiety: they listened to Bernard Herrmann’s Psycho score. Compare this to the stress of micromanagement, yearly review season, project bonuses and deadlines, or even general water cooler politics, and it’s no surprise what goes on in most corporate offices.

I’ve written before about how a manager’s primary job is to build trust and this is a good, concrete example why.

It’s also a good example of why it’s a company’s job to remove anxiety-causing policies. The less anxiety you cause your employees, the healthier they are and the healthier your culture and company is (we want people working together, not behaving selfishly). These policies include:

  • Annual performance reviews. Much has been written about this.
  • Individual performance-based bonuses. They have been proven to be counter-productive, without a single shred of evidence supporting their utility.
  • Limiting career and salary growth based on positions. People should not compete for a single “senior” spot.
  • Limiting PTO and not having separate sick days. Being sick is not a vacation.
  • Not forcing/encouraging people to take a vacation. This causes paranoia, burnout, and envy.
  • Limiting the flow of information. People will worry if they know what they need to know.

The list goes on and on. And the lesson is very simple:

When you reduce anxiety, you get better work.

1 Comment

Could employees choose their own manager?

by Rob Galanakis on 10/02/2015

Someone once brought up to me a plan about enabling employees to choose their own manager. The idea has stuck with me for a while, and being in my current position of authority I’ve pondered it more actively. I’ll use this post to collect my thoughts, and maybe present some ideas for discussion. I’m not going to evaluate the benefits or if this is a good idea, but only think about the practicalities.

First, let’s define the role of “manager”. There are many ways the role of manager can be split up or changed or redefined, but I’m specifically going to talk about the extremely popular and stubborn setup of Dev->Manager->Director->VP->CO, or whatever similar hierarchy. I do believe a better structure exists (or that the lack of structure is better!), but I have not seen it, and this arrangement is certainly the most popular, so let’s work from that.

Second, let’s define the manager’s responsibilities. There is the leadership aspect (setting direction for a team/group), and there is the procedural aspect (hiring, firing, raises). These can be found in the same person, or separate. If we operate in a strict hierarchy, where everyone in a team reports to a team’s lead, leadership and procedure must be handled by the same person. If people report to a “department manager” or someone else who is not a team, leadership and procedure are handled by different people.

That established, how would “choosing your manager” work?

“Choosing your manager” would mean individually choosing only the procedural person. The leadership person must be chosen collectively. The reasons for this are obvious. They could be the same person, though.

Collectively choosing the leader but having an assigned procedural manager will not work. The person doing the hiring, firing, and appraising ends up with the power anyway. It’s not fair to the leader and results in terrible politics when things are not in perfect alignment.

Choosing managers would basically prohibit outside hiring at management level. Good managers want to continue to be managers, and while I do believe they need to be talented programmers/designers/whatever, many would not want a 100% full time role with only the hope/possibility of doing management later. So you have cut down your pool of experienced managers significantly. But maybe that’s ok (most managers aren’t very good anyway).

The procedural stuff involves significant confidentiality in a traditional company. Generally, a manager is first and foremost vetted by management, and only secondarily by her reports. To flip this on its head would require radical transparency. Anyone at any time could become a “manager” and have access to extra confidential information. Salaries, at the very least, would need to be common knowledge (which would imply many other things are common knowledge).

Transparency is the big issue. Employees choosing their own managers would require a radically non-traditional company. At this extreme, you may as well get rid of the notion of managers altogether. Unfortunately it’s traditional companies that would benefit most from structural management changes. But at this point, I’m skeptical of “employees choosing their own manager” being a good idea. The thinking is in the right place! But I don’t see the way there. I’d love to hear of companies where this is done, though, as some concrete examples would help validate or disprove these thoughts (Spotify is the closest I can think of).

2 Comments

Automated testing shows a respect for employees

by Rob Galanakis on 1/02/2015

In the tech-artists.org G+ community page there was a comment on a thread about unit testing:

A key factor in TA tools is the speed at which we need to deliver them, and our audience is considerably smaller than, say, engine tools code. Therefor it becomes somewhat hard to justify the time spent on writing the unit tests, and then maintaining them as the tools change or are ported or updated to match new APIs.

In other words: Testing is great, but we don’t have time for it. Or the common alternative: Testing is great, but it’s not feasible to test what we’re doing.

Codebases without tests manifest themselves in teams that are stressed and overworked due to an ever-increasing workload and firefighting. Velocity goes down over time. Meanwhile, I’ve never known a team with thorough test coverage that delivered slower than a team without automated tests. In fact I’ve observed teams that had no tests and crunched constantly, added tests and became predictable and successful, then removed the tests after idiotic leadership decisions to artificially increase velocity, and watched their velocity drop way down once the testing infrastructure, and especially culture, fell into disrepair.

Companies that do not require automated tests do not respect their employees, and do not care about the long-term health of the company. It’s that simple (or they are incompetent, which is equally likely). We know that no testing results in stress, overwork, and reduced quality. We know that more testing results in more predictability, higher quality, and happier teams. I would love to blame management, but I see this nonchalant attitude about testing just as often among developers.

The “do it fast without tests, or do it slow with tests” attitude is not just wrong, but poisonous. You are going to be the one dealing with your technical debt. You are the bottleneck on call because your stuff breaks. You are the one who doesn’t get to work on new stuff because you spend all your time maintaining your old crap. You are the one who is crunching to tread water on velocity.

I have a simple rule: I will not work at a job that doesn’t have automated testing (or would be in any way inhibited instituting it as the first order of business).

  • I have this rule because I love myself and my family. There are enough unavoidable opportunities to interrupt evenings and weekends for work reasons. It is irresponsible to add more ways for things to break.
  • I have this rule because I care for the people I work with. I want them to have the same option for work-life balance, and work with me for a long time.
  • I have this rule because I want the company I’ve decided to invest in (employment is the most profound investment!) to be successful in the long term. Not until the end of the quarter, or even until I leave, but for a long, long time.
4 Comments

How do you estimate that which you’ve never done?

by Rob Galanakis on 29/01/2015

Have you heard about #noestimates? No? Well I’m sure you can guess what it is anyway. But reading the debates reminded me of a story.

While at Game Developer’s Conference a few years ago, I was arguing about estimation with a certain project manager, who, despite having no actual development experience, was in charge of development (Icelandic society is notoriously nepotistic).

“So, maybe no estimation works for your small projects, but when you have to do big projects, and you need to ask for budget, and coordinate many departments and offices, and you need to plan all this in advance, what do you do? How would you plan Incarna?”

Incarna was CCP’s expansion that introduced avatar/character-based “gameplay” into EVE Online. What shipped was the ability for your avatar to walk (not run!) around a room. It was massively over budget, behind schedule, and under delivered. A few months later, 20% of the company was laid off. There’s been no active development on Incarna since 2011, and World of Darkness- which continued to use Incarna’s core technology- was cancelled and the team laid off earlier this year. It was, quite simply, the biggest disaster I’ve seen or heard of my career.*

A character-based game is also something CCP had never done before. They are massively- MASSIVELY- more technologically complex than the “marbles in viscous fluid” EVE flight simulator. CCP did not have the in house experience, especially in Iceland, where most of the (very smart) engineering team had never worked on character based games.

So it was pretty hi-larious that a project manager was using Incarna as an example of why estimation is necessary. But cognitive dissonance is nothing new. Anyway, my response was:

“You don’t plan Incarna. You greenlight a month of development. At the end of a month, you see where things are. Do you keep going for another month? If you are happy with the spend and progress, keep going. If not, pull the plug. Once you can make a prediction at the start of a month, and it holds true for that month, and you do this two times in a row, maybe make a prediction for two months and see how it plays out.
You may pass a year this way. Well, a year isn’t a long time for developing a character-based MMO and game engine from scratch. But at the end of the year, you at least have some experience. But you keep going. If your velocity is consistently predictable, you estimate further out. Eventually, if you can get your velocity stable at the same time you’re growing and developing, you have a fighting chance.**
When your velocity isn’t stable, you reign things in and figure out. If you go through a year of missed month-long predictions, you need to change things drastically (or reboot entirely) if you hope to get something predictable.”

Nothing really insightful there of course- I’m just parroting what has worked me me and many, many others, from Lean-inspired methodologies (and this one in particular says traditional yearly budget cycles are responsible for many terrible business decisions).

A couple months ago I was asked if a significant new feature could get done by June. It would build on several months of foundation and other features. I responded that I was pretty confident that if we aim for June we would have it by September. My rationale, simply, was that previous similar projects shipped 3 or more months late, and I didn’t have enough experience with the team to give a more accurate estimate.

The best predictor of future behavior is past behavior. You need to create historical data before you can extrapolate and plan.

The historical data also needs to be “meaningful.” That is a much more nuanced topic, though.


* It should go without saying that disasters the scale of Incarna are 100% at the hands of management.

** On Star Wars: The Old Republic, management took an interesting strategy of driving velocity into the ground so that while it was terrible individually, it was at least stable. They could then increase the number of people resources and predict, pretty reliably, when it could ship. The game ended up costing about $200 million (I suspect much more, actually), but it wouldn’t have shipped otherwise.

1 Comment

Change should be the ally of quality

by Rob Galanakis on 26/01/2015

In The Beauty of Testing, Steven Sinofsky writes:

…great testers understand one the cardinal rules of software engineering—- change is the enemy of quality.

This is not a cardinal rule. This is a outdated and obsolete mode of thinking. Change is how you discover great UX. Change is how you refactor and reduce technical debt. Change is how you incrementally improve both your product and code quality.

Maybe that’s too obvious, and clearly Sinofsky isn’t arguing for static software. More nuanced (and the rest of the piece provides that nuance) would be “change inevitably introduces bugs, and bugs reduce quality.”

This too I take issue with. Your codebase should be verifiably better after you fix a bug: you’ve found a shortcoming in your automated tests, so you add a test, and maybe refactor some stuff as well. Or, you’ve identified a bad experience, and can change it to be better in a controlled manner. A bug is an opportunity for improvement. Without bugs, it can be very difficult to improve.*

It can be difficult for anyone who hasn’t worked in a codebase with extensive testing to understand this. In most cases, fixing bugs is playing whack-a-mole. Whack-a-mole is unacceptable to me. Every change we make at Cozy is making the code clearer, simpler, better tested. It’s making the product smoother, faster, and more intuitive.

Change is necessary; it is up to you to determine if it is a friend or foe.


If you’re practicing disciplined development and automated testing and not creating many bugs, good job! This post isn’t for you :)

4 Comments

Technical debt metaphors get it so wrong

by Rob Galanakis on 21/01/2015

In my previous post about technical debt, I explained how modern definitions of technical debt are harmful. Now I turn my attention to equally harmful metaphors.

Viktoras Makauskas made the following metaphor in a comment on my last post. This is a pretty perfect stand-in for metaphors I’ve read in other articles that harmfully define technical debt.

Imagine your car gets a strange rattle. You go to your mechanic and he says, “it’s your exhaust pipe holder, you need to replace it, but it’s gonna take a while to order a part and ship it, so just park your car here and come back in a week”. You say “no, I have this weekend trip planned, is there something we can do now?”. They say “yeah, we’ll put a strap on it meanwhile, just drive a little more careful and it should hold, but make sure to come back and do a proper fix”. Mechanic charges you now, and then a bit later.

This seems sensible on first read. But upon closer inspection, it’s quite clear the roles here are totally wrong*:

  • The mechanic is the programmer (the role of the “expert”). Well, a mechanic may or may not see your car ever again. They do not have a vested interest in your choice. A mechanic’s relationship to a car is totally different from a programmer’s relationship to code.
  • “You” are the “business” (the role of the “stakeholder”). The metaphor assumes that if you are irresponsible, it only impacts you (it’s your car, your money, your time). This is a problem. A programmer is impacted by business decisions in a way a mechanic is not impacted by whether you fix your car now or later.

This isn’t a simple language problem. It is a fundamental misunderstanding of roles that is naive to the way software development works. Programmers will be the primary sufferers of technical debt. Eventually the business will suffer with a slower pace of innovation and development and higher turnover. But well before that, programmers will be fixing (and refixing) obscure bugs, will bristle under management that tells them to go faster, will be working extra hours to try to improve things, and will eventually burn out. The business will only suffer once real damage has been done to a programming team, and many have given up.

This is why control of technical debt must be in the hands of programmers. Definitions or metaphors that urge otherwise are actively harmful.

Let me close by pointing out I’m just repeating what Ward Cunningham has already written about the original technical debt metaphor. The article ends with:

A lot of bloggers at least have explained the debt metaphor and confused it, I think, with the idea that you could write code poorly with the intention of doing a good job later and thinking that that was the primary source of debt.
I’m never in favor of writing code poorly, but I am in favor of writing code to reflect your current understanding of a problem even if that understanding is partial.

Thanks Ward.


* There are also a couple other problems with this metaphor. First, if “you” and the mechanic are the same person, and responsible for both business and implementation? In that case, there’s no need for a metaphor at all. Second, what happens if the exhaust fails? Do you become stranded? Does the car catch fire? What’s presented here is a false choice between a “correct” solution (replacement) or a “sloppy” solution (strapping it on). Why not rent a car? If there’s no responsible-but-relatively-cheap decision (there almost always is!), it’s still never acceptable to make an irresponsible decision.

4 Comments

Building Sphinx documentation for unfriendly code

by Rob Galanakis on 11/01/2015

Some Twitter friends were discussing how to get Sphinx to work with mayapy to build documentation for code that runs in Autodesk Maya. I’ve had to do this sort of thing extensively, for both Maya and editor/game code, and have even run an in-house Read The Docs server to host everything. I’ve learned a number of important lessons, but most relevant here is:

Always generate your documentation using vanilla Python. Never a custom interpreter.

There’s no philosophical reason for this*. I’ve just found it, by far, the path of least resistance. All you have to do is some mocking in conf.py:

import mock
for mod in ['maya.cmds', 'pymel.core']: # and whatever else you need
    sys.modules[mod] = mock.MagicMock()

(I do not have the code in front of me so this may be slightly wrong. Perhaps an ex-colleague from CCP can check what used to be in our conf.py.)

Now when Sphinx tries to import your module that has import pymel.core as pmc, it will work fine. That is, assuming your modules do not have some nasty side effects or logic on import requiring correctly functioning modules, which you should definitely avoid and is always unnecessary.

When your documentation generation breaks, it’s now a simple matter of adding a string in one place, rather than a several hour debugging session.

Don’t say I didn’t warn you!


* If anything, I’m philosophically more inclined to use mayapy. So that should tell you what sort of bogeymen await!

No Comments

Undefining “technical debt”

by Rob Galanakis on 6/01/2015

For me, technical debt is defined pretty loosely as stuff you don’t like in the code and need to change to keep up velocity. However, I’ve seen lots of articles lately discussing a precise definition of “technical debt.” I would sum them up as:

  • Technical debt is incurred intentionally. Sloppy code or bad architecture is not debt.
  • It is a business decision to incur technical debt.
  • It is a business decision to pay down technical debt.

I hate this characterization of technical debt. I hate it because it’s damaging. It assumes a conversation like this happens:

Manager: “How long to do this feature?”
Programmer: “We can do that feature in 4 weeks properly, or 2 weeks if we take shortcuts that will hurt our velocity in the future.”
Manager: “OK, take a shortcut and get it down ASAP.”
… 2 weeks later …
Manager: “How long to do this feature?”
Programmer: “We must spend 2 weeks paying down our technical debt, then another 2 weeks to do the feature.”
Manager: “That sounds fine.”

Every muscle in my body twinges when I think about this. Quality is not something you can put off to later. The idea that a team would do a sloppy job but have the rigor to repay it later is unbelievable. The closest I’ve seen is rewriting a system after years of shortcuts, which often does not end well. This mentality goes along with “how many bugs you have should be a business decision”. This isn’t OK. Do not write something you do not plan on living with. Do not place the responsibility of doing a good job on the business. I find it sad that a programmer would think such behavior acceptable. This is your life. This is your code. Take some responsibility. Take pride in your work.

Or don’t, and sling garbage while getting paid a pretty penny. Just don’t pretend you’re respecting your craft.


(I just want to take a moment to give credit to the team at Cozy. We recently had a couple weeks of crunch. The team delivered fully tested code the entire time).

10 Comments