Archive of articles classified as' "All Posts"

Back home

The manager’s responsibility to review code

9/04/2014

I believe any technical leader has a responsibility to review all the code that goes into a codebase.* I am certainly not the only person to feel this way (Joe Duffy as MSFT and Aras Pranckevičius as Unity have said the same).

Furthermore, I don’t believe the responsibility to review code ends at a certain level. Everyone from an Engineering Manager to the CTO should be reviewing code as well. In my experience, I’m able to do thorough reviews for 5 to 8 people, and more cursory reviews for another 15 to 20.

Code review above a team lead level** is not about micro-management. A manager should never, ever be saying “we do not use lambdas here, use a named function instead.” Instead, try “do you think this would be more readable with a named function instead of a lambda?” Or maybe you say nothing, and observe what happens, and inspect what other code looks like. If lambdas are common in the codebase, either your opinions need more information, or you have done a poor job educating.

Code reviews by managers should be about getting enough information to manage and lead effectively.*** It keeps you up to speed about what is going on, not just in terms of tasks, but in terms of culture. Are people writing spaghetti? Are bad choices challenged? Are hacks put in? Is code documented? Are standard libraries being used? Are the other technical leads reviewing and leading their teams effectively? You can learn an incredible amount through code review, and you need this information to do your job of leadership and management effectively.


*: I believe all programming managers and leaders must be able to program. I find it shameful this needs to be said.

**: It should go without saying, but team leads should be reviewing every checkin on that team.

**: Code reviews are the *genchi genbutsu, or the go and see part of Lean management.

2 Comments

Why Agile became meaningless

6/04/2014

Uncle Bob recently wrote a post about The True Corruption of Agile. I think it will be a defining post for me because, as I’ll explain in my next post, I’m ready to give up on Agile. It has become meaningless due to the corruption Uncle Bob describes, and trying to reclaim Agile isn’t possible.

Imagine the Lean movement without Toyota. Toyota is the guiding force in Lean because it grew out the The Toyota Way.* When Lean goes awry, Toyota- the company and its principles, practices, and culture- is there to set things straight.

Toyota can guide Lean because the company has been successful for decades and Toyota attributes its success to the principles and practices known as The Toyota Way. But for many years, Toyota’s success was explained away by anything except the Toyota principles. Finally, all that was left was The Toyota Way. Toyota is the Lean reference implementation.

Agile has no such entity. Instead, we have hundreds of “Agile” shops who attribute success to some (non-)Agile practices. Then, once they’ve evangelized their (non-)Agile stories, reality catches up with them and the success disappears.** But no one hears anything about that failure. The corruption and perversion here is inevitable.

Without a company like Toyota giving birth to Agile and showing others how to do it right, Agile was destined to become what it is now: meaningless and corrupt.


*: The Toyota Way started out as the Toyota Production System. They aren’t technically the same but for the purposes of this post there’s no reason to distinguish.

**: For example, maybe InnoTech decides to use Scrum on a global scale to ship an ambitious product, and talks a lot about how they pulled this off and what benefits it yielded. Years later, velocity is in the toilet because the endless mountains of technical debt created, and maybe the company has had layoffs. The Scrum transformation will be in a book or on a stage. The layoffs or technical debt will not.

1 Comment

Global Glob

4/04/2014

I am cleaning out my drafts and found this two year old post titled “Globals Glob” with no content. The story is worth telling so here we go.

There was a class the EVE client used to control how missiles behaved. We needed to start using it in our tools for authoring missiles with their effects and audio. The class and module was definitely only designed (I used the term loosely) to run with a full client, and not inside of our tools, which are vanilla Python with a handful of modules available.

My solution was the GlobalsGlob base class, which was just a bag of every singleton or piece of data used by the client that was unavailable to our tools. So instead of:

serviceManager.GetService('someservice').DoThing(self.id)

it’d be:

self.globalsGlob.DoThing(self.id)

The ClientGlobalsGlob called the service, but FakeGlobalsGlob did nothing. The GlobalsGlob allowed us to use the missile code without having to rewrite it. A rewrite was out of the question, as it had just been rewritten, except using the same code. (sigh)

Unsurprisingly, GlobalsGlob was super-fragile. So we added a test to make sure the interface between the client and fake globs were the same, using the inspect module. This helped, but of course things kept breaking.

This all continued until the inevitable and total breakdown of the code. Trying to use the missile code in tools was abandoned (I think it was, I have no idea what state it’s in). This was okay though, as we weren’t using the missile tools much after those few months. GlobalsGlob served its purpose, but I will never be able to decide if it was a success or failure.

No Comments

What does your Product Owner own?

3/04/2014

In a previous post, I came down hard on Agile leaders that don’t program. Now I’ll turn my sights to another part of the Scrum trinity: the Product Owner. I’ll raise some concerns for what I’ve seen it become in videogames, and suggestions for improving how we use the role.

Most product owners I’ve seen in the videogame industry are much closer to project managers than business owners. Their primary job is often the coordination, planning, and prioritization of the cross-team dependencies that the scaled-up nature of game development tends to create. I’ve seen designers and business/marketing in the PO role on occasion. It has sometimes gone very poorly.

I’ve always thought this situation strange, as the PO role most closely aligns with someone from the discipline of game design. We usually don’t have a problem with mapping a Creative Director or other core vision holder to the role of PO. After all, they are the product champion, and marry game design and business sense. A project manager clearly wouldn’t suffice here. But then other POs on the same project are all project managers. What gives?

There’s are some litmus tests for seeing how product ownership works in your organization.:

  • Do people “graduate” from Scrum Master to Product Owner?
  • Do the same people occupy both Scrum Master and Product Owner roles, concurrently or not?
  • Is your product owner leading and championing, or following orders (from above and from the team) and focused on execution (metrics, tracking)?

The skills between product owner and project manager are significantly different. There’s a problem if most people are seen as able to do both, and if POs aren’t coming primarily from design, business, and marketing.

There are lots of reasons things get this way. The important thing is to realize that the term PO isn’t a good fit for what most POs are doing. I see two options.

The first option is to commit to a Chief Product Owner/Area Product Owners structure (described in the footnotes*). Here, product ownership is seen as a distinct set of skills that bridge the business and design/creative side of development. If you have the right people (for example, POs for the overall/creative, technological, visual, and operational parts of the product), this can work quite well. I’d say this is a much better option, but frankly can be difficult or impossible to pull off if you do not have the right people or mindset.

The second option is to commit to having a single Product Owner, and having a project manager (Producer) on each team who is responsible for traditional project management duties and being a proxy for the real PO. They make few decisions of their own, but just act as dutiful middlemen. Usually the Producer will also take the role of Scrum Master, though I think this is a shame as their focus will be on traditional project management. This will make it difficult to make sure your teams are getting an ongoing Lean and Agile education.

Ultimately, the key is to acknowledge how product ownership in your organization works. If how people are fulfilling the role of PO does not seem to align with the literature, change something. You can choose option one, and change your organization to match the literature. Or you can choose option two, to abandon the literature, and find something that will work instead. In either case, do not continue the dissonance.

The core of Lean and Agile is continual improvement. If you are using confusing or inappropriate terms and organizational structures, you sow confusion. If you are confused and without direction, you cannot reliably improve.


*: Scaling Lean and Agile Development is the best book I’ve read about scaling Agile development methodologies. Regarding the role of the product owner, their recommendation is to have a single PO if possible, but to have a single Chief Product Owner and several Area Product Owners if one PO is impractical (which it often is in game development). Importantly, POs are tied to areas of the product, and not to teams (who can and should drift between areas of the product).

4 Comments

Mike Bland’s profound analysis of the Apple SSL bug

1/04/2014

Mike Bland has done a series of articles and presentations about the Apple SSL bug over on his blog. To be frank, it’s pretty much the only good coverage of the bug I’ve seen. He’s submitted an article to the ACM; crossing my fingers it gets printed, because it’s an excellent analysis. I’ve been frustrated it hasn’t gotten more traction.

Mike’s take is that this bug could have easily been caught with simple unit testing. It’s sad this has been so overlooked. We gobble up the explanations that do not force us to confront our bad habits. We can point and laugh at the goto fail (“hahaha, never use goto!”), or just shrug at the merge fail (“we’ve all mistakes merging”). It’s much more difficult to wrestle with the fact that this error- like most- is because we are lazy about duplicating code and writing tests.

I have no problem categorically saying anyone who doesn’t blame the SSL bug on lack of automated testing and engineering rigor is making excuses. Not for Apple but for his or herself.

No Comments

Configuration in Host the Docs

28/03/2014

With Host the Docs, I chose to eschew configuration files and use a sphinx-style “conf.py” approach (I have previously written about how I like this approach). If a conf.py file is found, it is used for configuration, allowing it to override default configuration options. I also allow configuration through environment variables, where the environment variable name is the same as the conf.py variable but with HTD_ prepended.

So for example, in hostthedocs/getconfig.py, I have code like the following:

try:
    import conf
except ImportError:
    conf = None

def get(attr, default):
    result = os.getenv('HTD_' + attr.upper())
    if result is None:
        result = getattr(conf, attr, None)
    if result is None:
        result = default
    return result

welcome = get('welcome', 'Welcome to Host the Docs!')
# Other configuration values

I was initially going to use a json or yaml file, but always find their contracts a bit unclear, especially when paths are involved in the configuration (if the config uses relative paths, what are the paths relative to?). It also involves more code.

I was really happy with how using Python for configuration worked out in Host the Docs.

1 Comment

“What did you learn?”

25/03/2014

When something bad happens to someone (firing, demotion, bad review, big failure), it’s natural for managers to ask that person “what did you learn?*

Unfortunately the answer is rarely what a manager wants to hear, and it’s also largely useless.**

Asking the question phrases it as the employee’s problem, while theory and experience both tell us that it’s far more often the management that is at primary fault (work environment, culture, all sorts of common cause variation. It is not at all useful to ask the employee “what did you learn?” unless the goal of the question is a) pure personal curiousity, as when family/friends/coworkers ask, or b) to get the employee’s take on how to improve the system so these things don’t happen.

Are we masters of our own destiny? I find thinking so is a useful way to behave personally, but a naive or dishonest way to lead and manage. If you believe that an employee is the primary driver of their behavior, rather than the system he or she works in, then you’re probably relying on destiny to create a successful company. Good luck! You may also consider playing the lottery.***

Unless you can reliably improve and grow the system and culture, you are relying on luck, timing, and personal attributes to create a successful organization. That’s fine, but very few managers would admit to wanting such a company. An event that disrupts or upsets an employee is a great opportunity to learn, so here are some better questions to ask your employee when it happens. (for clarity, I will use “we” for management and “you” for employee)

Better questions to ask your employee

What can we learn from this? I assume we think highly of our employee (we hired them****, or at least haven’t fired them before this). They probably have the best view of what went right and wrong. And if not the best view, then at least a unique one worth hearing. We have as much to learn as they do. After all, we- the manager, employee, and probably many other connected people- failed.

What do you think caused this? Your employee fell victim to the system. The difficult part is figuring out what parts of the system were the aggressors. Remember, even if this were truly their fault, our organization can’t grow from finding the employee at fault, so it behooves us to find something we can learn from.

Were you surprised? When people take risks, they expect to lose part of the time. The employee’s failure may not be due to a mistake, but a calculated risk that didn’t work out. This fundamentally changes what the “learning” is about. Let’s not presume bad outcomes were due to a lack of understanding or miscalculation on the part of the employee, but rather assume bad outcomes are due to a shortcoming in our management and the system.

Why were we surprised at the outcome? Why didn’t we see this coming? If we did see it coming, why didn’t we act earlier? Maybe your employee can help you learn to see this happening earlier next time, so you can do something about it. Or maybe you were warned about it, but saw what you wanted to see, and your employee can help you realize that.

What would “success” have looked like? Some situations are “unwinnable.” There war is lost, but the employee still believes there is hope. Find out what they were fighting for. You can make sure others that are fighting for the same cause don’t meet the same fate, either because you fix the issue, or do a better job explaining it.

There are many other good questions that should be asked. Big “failures” are great opportunities to learn, so let the discussion flow. You won’t learn and grow as an organization if your default response is to blame the employee. Every time someone gets a bad review, is fired, quits, or royally messes up, we must use that opportunity to improve as an organization.


*: I will state so there’s no confusion: I’m not talking about the words themselves. I’m talking about the idea that you are asking the employee what they learned as the primary way to involve them in the retrospection of “their” significant failure.

**: Usually what the employee probably thinks is you’re an asshole, the culture is broken, and the organization is fucked. So I spend this article focusing on why the question is useless.

***: I don’t worry about offending people here, because no one ever thinks they’re a bad manager. And you certainly aren’t!

****: I use ‘they’ or ‘their’ instead of “he or she” or “s/he” or whatever. I’m not sure what is in vogue today.

10 Comments

Introducing Host The Docs

22/03/2014

A month or so ago I created Host the Docs, and it’s been quietly running at work some success, so I figure it’s time to talk about it. The code is on GitHub at https://github.com/rgalanakis/hostthedocs, and here’s a description from the readme:

Host the Docs is a simple way to host static code documentation. Its main use is as a self-hosted server for your organization’s private documentation. Better alternatives are available for open source projects, such as Read the Docs or Github Pages.

Host the Docs was created after a long day of banging my head against the wall trying to get Read the Docs set up with private GitHub repositories, and having helped develop a plugin to get it to work with Perforce previously. What the world needed was a way to easily host documentation from several projects, from any source or language or SCM.

Seriously, let other people generate their own docs, I just want to Host the Docs!

It’s super easy to get set up, and requires no database. You can even just serve everything through Flask, if you don’t want to set up a webserver like Apache or nginx. It’s pretty well tested and configurable and thoroughly documented. We had it running on a new Linux VM in just a few minutes, and use it to host HTML documentation for Python and C# projects (we used Sphinx to generate the Python docs and Doxygen for C#).

I encourage you to try it out if you are interested in painless documentation hosting! I’m also very open to fixes/changes/pull requests, as long as it doesn’t overcomplicate things.

No Comments

Being amazed by software development

1/03/2014

I am continually amazed by the state of software development. I am amazed at how broken things seem to be, and I’m amazed at what powerful tools we have to fix things.

A few weeks ago, I struggled to find a documentation hosting solution. We have an internal version of Read the Docs which took far too much effort to get set up (dependency problems, creating a new SCM backend) and administrate, but Read the Docs doesn’t work with private Github Repos. After some hacking, we still couldn’t get it to work. So I looked at hasdocs.com, which promised to host our docs in the cloud, but was totally broken. I looked around at other solutions and was tearing my hair out. I was amazed that no good solutions already existed, and was more amazed that the best solutions are somewhere on the spectrum of broken (no offense here to RtD).

At the end of the day, it occurred to me my use case was incredibly simple: just provide an index page for (already generated) HTML documentation. Let people POST their HTML files and some metadata, and just dynamically generate the index based on what’s available. I was going to run this behind a firewall, so I didn’t need to worry about security. Hell I could even get away with no database, and just read the filesystem.

The next day (a Saturday), while my son and wife napped, Host the Docs was born. It took 1.5 hours to create an initial working version, and then I spent a few hours here and there polishing things up. It was painless to deploy, and I did some more improvements after that. Throughout this experience, a few things struck me:

  • I’m amazed by frameworks like Flask and Bootstrap. You can create a reasonable site in no time that is totally maintainable. At no point was I “hacking” to get something up and running, it was instead a very small first version I was able to iterate on.
  • I’m amazed by Linux. I have to use Windows at work, but feel like I develop faster on my Linux Mint netbook as I do on my Windows 7 workstation. The power at my fingertips is divine. It makes me mad at Windows.
  • I’m amazed how well some software works the same way I’m amazed at how broken some software is. Software is truly evolving; for every Flask, there are confusing and broken web frameworks.
  • I’m amazed how much time having autonomous teams can save. It would have taken a day or more to deploy HTD if I had to go through IT to provision a normal virtual machine (I doubt that frustration is unique in our industry). Instead, by giving teams an AWS budget and not centrally controlling things, HTD was live in minutes.

I’ll post more fully about Host the Docs later. I just wanted to express my satisfaction before it wore off :)


(BTW, I’m aware how similar this sentiment is to Jeff Knupp’s story of building Bull: Python and Flask are ridiculously powerful. Probably not coincidentally it is also a story of using Flask :)

6 Comments

Agile project management versus agile development

24/02/2014

I have a saying I like to use when discussing Scrum: Scrum is an Agile project management methodology, not an Agile development methodology. Scrum delivers tools for managing the project (planning, scheduling), but very little for how to develop (design, program, test) it. To do Agile properly, you really need both. This is why eXtreme Programming (XP) and Scrum “fit together” so nicely, with XP telling you how to build your product and Scrum usually taking a higher-level view (and the overlaps are usually the same practices).

The complimentary nature of project management and development methodologies is important to understand and embrace. It is also one reason I don’t believe you can implement Scrum effectively without a programming background. Ultimately this comes down to a very simple thing for me:

If you do not have comprehensive automated tests, you cannot be Agile.(1) I consider this a fundamental truth. Yet rarely do books about Scrum spend more than a few pages talking about how vital this is, rarely is it discussed enough in Scrum Master training courses, rarely do the project managers now running most Agile implementations seem to understand this.

Automated tests are the fundamental building blocks from which all other Agile practices flow. This also informs how I treat Agile as a whole: Agile project management should be free-flowing and not rigid, but Agile development should be rigorously adhered to. That is sure to be inflammatory so let me elaborate.

Rigorous Agile development

  1. If automated tests are fundamental, and Test Driven Development (TDD) is the only way to get comprehensive test coverage, that means TDD is not optional. You don’t need to use it everywhere, but TDD should be the rule, not the exception.
  2. In the same way, you cannot be Agile if one person ends up being a bottleneck, so collective code ownership is required. I think it’s less important whether this is done through pairing or code reviews (probably both), and more that collective code ownership is a rule and any exceptions are seen as a big problem. Automated tests are required to collectively own code, as it’s the only way people can reliably make changes.
  3. Continuous integration is equally important, and depends on automated tests to be reliable and problems to be quickly fixed (it relies on the other two practices). You must always have a high quality build available and the code people are writing should be quickly (once a day or more) and easily getting into that build.

These three practices I consider absolute.(2). Maybe you can add some, but you are not allowed to decide you want to exclude any of those three from your Agile implementation.(3) To do so invalidates the principles on which Agile rests. So it follows that you should not be allowed to have zero experience in them if you’re an Agile leader. You cannot be Agile without them, no matter how briefly Scrum literature covers these topics; I would bet most of the writers of that literature would agree.

Flexible Agile project management

Opposite the rigorous adherence to specific development practices is experimentation with general project management practices. In this area, things are much more about principles (primarily feedback and continuous improvement) and less about practices or processes. Your sprint should be as long as you need to get good feedback, which varies depending on project/team/technology. Your retrospectives should be run so you can continuously improve. Your planning should be so you can get more accurate and insightful. Ten solutions can work well at ten different workplaces. Even more, ten solutions can work well at the same workplace(4). Just make sure you are continually improving, and keep trying new things!

Having your cake and eating it too

This distinction between development and project management is how I navigate the rift between Agile nihilists and Agile purists. The former say with disdain, “Whatever works for you!” The latter chant fervently, “You cannot pick and choose!” It turns out they can both be right, but it’s important to understand how and why. The nihilists end up floating, never realizing the transformative power of Agile because they refuse to adhere to the three vital, but initially taxing, processes. The purists can drive change, but not transform, because they do not create new practices that fully embrace each unique situation.(5) Rather than trying for a happy medium between the poles, I find Agile is best done by being at the extremes simultaneously.


  1. I don’t know how to rebut arguments against this point, other than asking “have you worked on a project that was well-developed with TDD?” If not, I would try it out before you make excuses for a compromised form of Agile without comprehensive automated tests.
  2. More accurately, what they achieve I consider absolute. If you wanted to get rid of any of them, you’d need to replace them with something suitable. For example, if you didn’t want to use TDD, you’d need to demonstrate some other way to reliably build comprehensive automated tests. And actually I have great hopes for some automated alternative to TDD one day…
  3. Uncle Bob recently posted about how software projects need this sort of discipline, perhaps by having a “foreman”. I don’t agree on the solution, but I do agree that we do need to rigorously adhere to certain practices. http://blog.8thlight.com/uncle-bob/2014/02/21/WhereIsTheForeman.html
  4. This is probably attributable to the Hawthorn Effect: http://en.wikipedia.org/wiki/Hawthorne_effect
  5. To be fair, many purists (especially outside of programming) overlook the three vital development practices because they are so keen on implementing the easier Scrum project management tools that require less training and invasive changes.
5 Comments

Switch to our mobile site