Mike Bland’s profound analysis of the Apple SSL bug

by Rob Galanakis on 1/04/2014

Mike Bland has done a series of articles and presentations about the Apple SSL bug over on his blog. To be frank, it’s pretty much the only good coverage of the bug I’ve seen. He’s submitted an article to the ACM; crossing my fingers it gets printed, because it’s an excellent analysis. I’ve been frustrated it hasn’t gotten more traction.

Mike’s take is that this bug could have easily been caught with simple unit testing. It’s sad this has been so overlooked. We gobble up the explanations that do not force us to confront our bad habits. We can point and laugh at the goto fail (“hahaha, never use goto!”), or just shrug at the merge fail (“we’ve all mistakes merging”). It’s much more difficult to wrestle with the fact that this error- like most- is because we are lazy about duplicating code and writing tests.

I have no problem categorically saying anyone who doesn’t blame the SSL bug on lack of automated testing and engineering rigor is making excuses. Not for Apple but for his or herself.

No Comments

What if Carl Sagan were a hack?

by Rob Galanakis on 31/03/2014

I was watching the first episode of Cosmos, and Neil deGrasse Tyson talked some about how stellar of a scientists Carl Sagan was and what an impact Carl had on Neil personally. Carl’s abilities were important for his advocacy, because a) it lent him credibility, and b) it allowed him to engage. He practiced while he advocated. I can’t imagine Carl Sagan achieving the impact and legacy he did by abandoning the lab for the lecture circuit.

What a powerful lesson for those of us that manage people doing work we’ve never done. How can we deeply connect with them?

What a reminder for those of us that have moved into managing and left behind creating. Should our dues, once paid, last forever?

What a feeling for those of us who have moved into management out of expectation. Is it right to tell people what to do, while we have lost enough passion to do it ourselves?

2 Comments

Planet Mars, the simple feed aggregator

by Rob Galanakis on 30/03/2014

In December, I spent some time polishing up a new fork of the old PlanetPlanet Python feed aggregator, named Planet Mars: https://github.com/rgalanakis/planet-mars . I chose the name to contrast it with Planet Venus, a successful PlanetPlanet fork. My goal for Planet Mars is to make it as simple as possible, to be the best aggregator for the very common use case. It’s based on my experience running a PlanetPlanet-based river at http://tech-artists.org/planet. That feed has been powered by Planet Mars for the last few months.

The main features are:

  • Jinja2 templating, in addition to the original htmltmpl engine. Old templates still work, new templates can use a better language.
  • Parallelization of feed updating, so things are many times faster.
  • The code is stripped down and much easier to understand and improve.
  • Planet Mars can be installed via pip, so instead of having to place your customizations in its package directory, you can pip install the GitHub repo and import it as a module into your own code (you can see the Planet TechArt repo as an example).
  • Planet Mars otherwise very similar to PlanetPlanet, making switching to PlanetMars very easy :)

Please take a look and tell me what you think, especially if you are an original Planet user.I’m very open to improvements and suggestions. If there’s any demand, I can put this on PyPI as well.

3 Comments

Why I love blogging

by Rob Galanakis on 29/03/2014

I started to write a post about how I missed multithreading and speed going from C# to Python. I ended up realizing the service we built which inspired the post was poorly designed and took far more effort than it should have. The speed and multithreading of C# made it easier to come up with an inferior design.

The service needed to be super fast, but the legacy usage pattern was the problem. It needed to run as a service, but then we realize it’ll be running as a normal process. This is what happens when you focus too much on requirements and architecture instead of delivering value quickly. You create waste.

I wouldn’t have realized all of this if I didn’t sit down to write about it.

No Comments

Configuration in Host the Docs

by Rob Galanakis on 28/03/2014

With Host the Docs, I chose to eschew configuration files and use a sphinx-style “conf.py” approach (I have previously written about how I like this approach). If a conf.py file is found, it is used for configuration, allowing it to override default configuration options. I also allow configuration through environment variables, where the environment variable name is the same as the conf.py variable but with HTD_ prepended.

So for example, in hostthedocs/getconfig.py, I have code like the following:

try:
    import conf
except ImportError:
    conf = None

def get(attr, default):
    result = os.getenv('HTD_' + attr.upper())
    if result is None:
        result = getattr(conf, attr, None)
    if result is None:
        result = default
    return result

welcome = get('welcome', 'Welcome to Host the Docs!')
# Other configuration values

I was initially going to use a json or yaml file, but always find their contracts a bit unclear, especially when paths are involved in the configuration (if the config uses relative paths, what are the paths relative to?). It also involves more code.

I was really happy with how using Python for configuration worked out in Host the Docs.

1 Comment

“What did you learn?”

by Rob Galanakis on 25/03/2014

When something bad happens to someone (firing, demotion, bad review, big failure), it’s natural for managers to ask that person “what did you learn?*

Unfortunately the answer is rarely what a manager wants to hear, and it’s also largely useless.**

Asking the question phrases it as the employee’s problem, while theory and experience both tell us that it’s far more often the management that is at primary fault (work environment, culture, all sorts of common cause variation. It is not at all useful to ask the employee “what did you learn?” unless the goal of the question is a) pure personal curiousity, as when family/friends/coworkers ask, or b) to get the employee’s take on how to improve the system so these things don’t happen.

Are we masters of our own destiny? I find thinking so is a useful way to behave personally, but a naive or dishonest way to lead and manage. If you believe that an employee is the primary driver of their behavior, rather than the system he or she works in, then you’re probably relying on destiny to create a successful company. Good luck! You may also consider playing the lottery.***

Unless you can reliably improve and grow the system and culture, you are relying on luck, timing, and personal attributes to create a successful organization. That’s fine, but very few managers would admit to wanting such a company. An event that disrupts or upsets an employee is a great opportunity to learn, so here are some better questions to ask your employee when it happens. (for clarity, I will use “we” for management and “you” for employee)

Better questions to ask your employee

What can we learn from this? I assume we think highly of our employee (we hired them****, or at least haven’t fired them before this). They probably have the best view of what went right and wrong. And if not the best view, then at least a unique one worth hearing. We have as much to learn as they do. After all, we- the manager, employee, and probably many other connected people- failed.

What do you think caused this? Your employee fell victim to the system. The difficult part is figuring out what parts of the system were the aggressors. Remember, even if this were truly their fault, our organization can’t grow from finding the employee at fault, so it behooves us to find something we can learn from.

Were you surprised? When people take risks, they expect to lose part of the time. The employee’s failure may not be due to a mistake, but a calculated risk that didn’t work out. This fundamentally changes what the “learning” is about. Let’s not presume bad outcomes were due to a lack of understanding or miscalculation on the part of the employee, but rather assume bad outcomes are due to a shortcoming in our management and the system.

Why were we surprised at the outcome? Why didn’t we see this coming? If we did see it coming, why didn’t we act earlier? Maybe your employee can help you learn to see this happening earlier next time, so you can do something about it. Or maybe you were warned about it, but saw what you wanted to see, and your employee can help you realize that.

What would “success” have looked like? Some situations are “unwinnable.” There war is lost, but the employee still believes there is hope. Find out what they were fighting for. You can make sure others that are fighting for the same cause don’t meet the same fate, either because you fix the issue, or do a better job explaining it.

There are many other good questions that should be asked. Big “failures” are great opportunities to learn, so let the discussion flow. You won’t learn and grow as an organization if your default response is to blame the employee. Every time someone gets a bad review, is fired, quits, or royally messes up, we must use that opportunity to improve as an organization.


*: I will state so there’s no confusion: I’m not talking about the words themselves. I’m talking about the idea that you are asking the employee what they learned as the primary way to involve them in the retrospection of “their” significant failure.

**: Usually what the employee probably thinks is you’re an asshole, the culture is broken, and the organization is fucked. So I spend this article focusing on why the question is useless.

***: I don’t worry about offending people here, because no one ever thinks they’re a bad manager. And you certainly aren’t!

****: I use ‘they’ or ‘their’ instead of “he or she” or “s/he” or whatever. I’m not sure what is in vogue today.

10 Comments

Introducing Host The Docs

by Rob Galanakis on 22/03/2014

A month or so ago I created Host the Docs, and it’s been quietly running at work some success, so I figure it’s time to talk about it. The code is on GitHub at https://github.com/rgalanakis/hostthedocs, and here’s a description from the readme:

Host the Docs is a simple way to host static code documentation. Its main use is as a self-hosted server for your organization’s private documentation. Better alternatives are available for open source projects, such as Read the Docs or Github Pages.

Host the Docs was created after a long day of banging my head against the wall trying to get Read the Docs set up with private GitHub repositories, and having helped develop a plugin to get it to work with Perforce previously. What the world needed was a way to easily host documentation from several projects, from any source or language or SCM.

Seriously, let other people generate their own docs, I just want to Host the Docs!

It’s super easy to get set up, and requires no database. You can even just serve everything through Flask, if you don’t want to set up a webserver like Apache or nginx. It’s pretty well tested and configurable and thoroughly documented. We had it running on a new Linux VM in just a few minutes, and use it to host HTML documentation for Python and C# projects (we used Sphinx to generate the Python docs and Doxygen for C#).

I encourage you to try it out if you are interested in painless documentation hosting! I’m also very open to fixes/changes/pull requests, as long as it doesn’t overcomplicate things.

No Comments

Who is using PyPy?

by Rob Galanakis on 11/03/2014

I brought up the idea of using PyPy for an internal service we are building at work, and was asked what actual projects are using PyPy? I had no answer and couldn’t find one. Are there are well known projects using PyPy? Do you know of anyone that is using it to run small servers, even? I find PyPy hugely exciting, and I know it’s current limitations and issues (as well as its benefits), but I’m wondering if anyone’s used it yet and what their experience has been.

7 Comments

Being amazed by software development

by Rob Galanakis on 1/03/2014

I am continually amazed by the state of software development. I am amazed at how broken things seem to be, and I’m amazed at what powerful tools we have to fix things.

A few weeks ago, I struggled to find a documentation hosting solution. We have an internal version of Read the Docs which took far too much effort to get set up (dependency problems, creating a new SCM backend) and administrate, but Read the Docs doesn’t work with private Github Repos. After some hacking, we still couldn’t get it to work. So I looked at hasdocs.com, which promised to host our docs in the cloud, but was totally broken. I looked around at other solutions and was tearing my hair out. I was amazed that no good solutions already existed, and was more amazed that the best solutions are somewhere on the spectrum of broken (no offense here to RtD).

At the end of the day, it occurred to me my use case was incredibly simple: just provide an index page for (already generated) HTML documentation. Let people POST their HTML files and some metadata, and just dynamically generate the index based on what’s available. I was going to run this behind a firewall, so I didn’t need to worry about security. Hell I could even get away with no database, and just read the filesystem.

The next day (a Saturday), while my son and wife napped, Host the Docs was born. It took 1.5 hours to create an initial working version, and then I spent a few hours here and there polishing things up. It was painless to deploy, and I did some more improvements after that. Throughout this experience, a few things struck me:

  • I’m amazed by frameworks like Flask and Bootstrap. You can create a reasonable site in no time that is totally maintainable. At no point was I “hacking” to get something up and running, it was instead a very small first version I was able to iterate on.
  • I’m amazed by Linux. I have to use Windows at work, but feel like I develop faster on my Linux Mint netbook as I do on my Windows 7 workstation. The power at my fingertips is divine. It makes me mad at Windows.
  • I’m amazed how well some software works the same way I’m amazed at how broken some software is. Software is truly evolving; for every Flask, there are confusing and broken web frameworks.
  • I’m amazed how much time having autonomous teams can save. It would have taken a day or more to deploy HTD if I had to go through IT to provision a normal virtual machine (I doubt that frustration is unique in our industry). Instead, by giving teams an AWS budget and not centrally controlling things, HTD was live in minutes.

I’ll post more fully about Host the Docs later. I just wanted to express my satisfaction before it wore off :)


(BTW, I’m aware how similar this sentiment is to Jeff Knupp’s story of building Bull: Python and Flask are ridiculously powerful. Probably not coincidentally it is also a story of using Flask :)

6 Comments

Agile project management versus agile development

by Rob Galanakis on 24/02/2014

I have a saying I like to use when discussing Scrum: Scrum is an Agile project management methodology, not an Agile development methodology. Scrum delivers tools for managing the project (planning, scheduling), but very little for how to develop (design, program, test) it. To do Agile properly, you really need both. This is why eXtreme Programming (XP) and Scrum “fit together” so nicely, with XP telling you how to build your product and Scrum usually taking a higher-level view (and the overlaps are usually the same practices).

The complimentary nature of project management and development methodologies is important to understand and embrace. It is also one reason I don’t believe you can implement Scrum effectively without a programming background. Ultimately this comes down to a very simple thing for me:

If you do not have comprehensive automated tests, you cannot be Agile.(1) I consider this a fundamental truth. Yet rarely do books about Scrum spend more than a few pages talking about how vital this is, rarely is it discussed enough in Scrum Master training courses, rarely do the project managers now running most Agile implementations seem to understand this.

Automated tests are the fundamental building blocks from which all other Agile practices flow. This also informs how I treat Agile as a whole: Agile project management should be free-flowing and not rigid, but Agile development should be rigorously adhered to. That is sure to be inflammatory so let me elaborate.

Rigorous Agile development

  1. If automated tests are fundamental, and Test Driven Development (TDD) is the only way to get comprehensive test coverage, that means TDD is not optional. You don’t need to use it everywhere, but TDD should be the rule, not the exception.
  2. In the same way, you cannot be Agile if one person ends up being a bottleneck, so collective code ownership is required. I think it’s less important whether this is done through pairing or code reviews (probably both), and more that collective code ownership is a rule and any exceptions are seen as a big problem. Automated tests are required to collectively own code, as it’s the only way people can reliably make changes.
  3. Continuous integration is equally important, and depends on automated tests to be reliable and problems to be quickly fixed (it relies on the other two practices). You must always have a high quality build available and the code people are writing should be quickly (once a day or more) and easily getting into that build.

These three practices I consider absolute.(2). Maybe you can add some, but you are not allowed to decide you want to exclude any of those three from your Agile implementation.(3) To do so invalidates the principles on which Agile rests. So it follows that you should not be allowed to have zero experience in them if you’re an Agile leader. You cannot be Agile without them, no matter how briefly Scrum literature covers these topics; I would bet most of the writers of that literature would agree.

Flexible Agile project management

Opposite the rigorous adherence to specific development practices is experimentation with general project management practices. In this area, things are much more about principles (primarily feedback and continuous improvement) and less about practices or processes. Your sprint should be as long as you need to get good feedback, which varies depending on project/team/technology. Your retrospectives should be run so you can continuously improve. Your planning should be so you can get more accurate and insightful. Ten solutions can work well at ten different workplaces. Even more, ten solutions can work well at the same workplace(4). Just make sure you are continually improving, and keep trying new things!

Having your cake and eating it too

This distinction between development and project management is how I navigate the rift between Agile nihilists and Agile purists. The former say with disdain, “Whatever works for you!” The latter chant fervently, “You cannot pick and choose!” It turns out they can both be right, but it’s important to understand how and why. The nihilists end up floating, never realizing the transformative power of Agile because they refuse to adhere to the three vital, but initially taxing, processes. The purists can drive change, but not transform, because they do not create new practices that fully embrace each unique situation.(5) Rather than trying for a happy medium between the poles, I find Agile is best done by being at the extremes simultaneously.


  1. I don’t know how to rebut arguments against this point, other than asking “have you worked on a project that was well-developed with TDD?” If not, I would try it out before you make excuses for a compromised form of Agile without comprehensive automated tests.
  2. More accurately, what they achieve I consider absolute. If you wanted to get rid of any of them, you’d need to replace them with something suitable. For example, if you didn’t want to use TDD, you’d need to demonstrate some other way to reliably build comprehensive automated tests. And actually I have great hopes for some automated alternative to TDD one day…
  3. Uncle Bob recently posted about how software projects need this sort of discipline, perhaps by having a “foreman”. I don’t agree on the solution, but I do agree that we do need to rigorously adhere to certain practices. http://blog.8thlight.com/uncle-bob/2014/02/21/WhereIsTheForeman.html
  4. This is probably attributable to the Hawthorn Effect: http://en.wikipedia.org/wiki/Hawthorne_effect
  5. To be fair, many purists (especially outside of programming) overlook the three vital development practices because they are so keen on implementing the easier Scrum project management tools that require less training and invasive changes.
5 Comments