Reflections on porting Planet

by Rob Galanakis on 22/12/2013

I use the Python Planet software to generate the RSS feed and static site  for, like is done for I’ve been using it for several years and recently got the urge to do some open source work, so I “forked” the original Planet code (actually, the Bazaar repo is no longer so I just worked off the tar) and began to port it to use Jinja2 instead of the HTML templating engine derived from Perl. I am mostly done and just need to clean out a few more specific things before linking to the github repo here, but I did have some interesting reflections on porting code that is 7.5 years old!

  • All the dependencies were just copied in. This was before the popularity of pip. Was this a common way to do dependency management back then (how widely used was easy_install)?
  • In some cases it was difficult to know what was part of planet and what was copied in. I’ve replaced what I could with a requirements.txt. feedparser is still maintained, sanitize is kindly provided on PyPI, and the other modules were provided for way old compatibility, so now only htmltmpl is copied in.
  • The standard library is way better nowadays. This code ran on 2.1 or 2.2 (conflicting comments to which is the minimum). No socket.setdefaulttimeout, no logging module. I guess the backwards compatibility situation is even worse nowadays though, which code often having to support 2.5 to 3.4!
  • The code was still pretty good on every level. Structurally it is well thought out. Surprising to me was that stylistically it was still pretty good as well, aside from missing use of new language features (for example: context managers, ‘in’ keyword).
  • It did a lot of stuff, especially via configuration, that I can’t imagine anyone uses. In fact Planet is really an entire static site generator. Maybe people used it for that in 2006? Anyway, nowadays there are so many better options, Planet is only used for pages like Planet Python and Planet TechArt, so I took out unnecessary configuration which simplified things greatly. And of course I dropped the compatibility modules that are no longer needed, and the utility which I never used.
  • It had “unit” tests! And they used the unittest module! But I couldn’t get them to work (and they’re now entirely broken). It even had a script to run all of them, though nowadays I guess everyone uses test runners, thank goodness.
  • Looking over the HTML templating engine code wasn’t the first time I’ve felt uncomfortable reading code, to later find out it was ported from Perl. Perl systems stand out like a sore thumb! I was very happy to get Jinja2 working. I actually don’t have complaints with, but it isn’t what I want to be using in 2014.

I work every day in a Python codebase originally written in the early 2000′s, so I’m not stranger to legacy systems. But that is “enterprise” code which is different from open source packages, and I wasn’t writing Python in 2006, so it is interesting to get a glimpse into what the Python world was like 7.5 years ago. Kudos to the original authors for doing a great job.


Removing external hiring as a tool (Part 3 of 3)

by Rob Galanakis on 17/12/2013

In this post I hope to explain how hiring externally as a tool for fixing problems ultimately leads to a weaker organization.

When I began writing this post, I was having a hard time. Whereas the post talking about what a bad idea firing is was easy, the situation is considerably better for hiring. For starters, there are more organizations that do a good job. Very rigorous hiring practices, even during growth. It’s also easier to talk about how a company hires people than how it fires people, as its generally a positive experience (which is why this article probably seems a lot weaker than the previous). Of course sometimes you need to hire specialists for very specific areas (I’m talking something like ‘hardware emulation’ or ‘low-level rendering engines’, not ‘databases’ or ‘UI programming’), where it is prohibitive to train or grow people in a year. And sometimes there are amazing individuals you just need to have (take Ward Cunningham being hired into New Relic or John Carmack becoming CTO of Occulus as examples). Then there are organizations in periods of rapid growth. While rapid growth is always risky, it’s often necessary to bring enough force to bear in innovative companies. I’d much rather see stable growth but it’s not always an option. So I needed a clear demonstration of the problems.

Then, like manna from an ironic heaven, I saw this article about Abercrombie’s CEO. After investor calls for his resignation, Abercrombie renewed his contract and stated their plan to hire in three executives to manage each of their brands. The idea is that one of them will replace the current CEO, and the CEO has done a bad job managing things, and of course did not groom internal candidates, so it seems they are due to come from the outside.

I don’t understand how anyone expects this to end well. Each Brand President will come in, make changes (including, probably, layoffs!) that benefit the short term (because their goal is to be CEO), and then: 1) The one that becomes CEO will stay a few years. The average tenure of a CEO in America is about 4.5 years. 2) The two that do not will likely leave, meaning new executives will be hired in and there will be more instability.

This is the sort of hiring- not just at the CEO level but all leadership levels down to team lead- that I think can be misguided. Why?

Mostly, because it disguises a problem. Most organizations buy into the idea that internal candidates should be preferred to external ones (another Lean principle!), yet still need to look outside for senior talent and managers. I would compare the situation and solution to DevOps: if deployments are an issue, the worst thing to do is isolate their handling to a small group and deploy less frequently. The DevOps movement has shown us the power of the mantra of “if it hurts, do it more often.”

I believe the inability of an organization to groom internal candidates indicates severe management problems, and because the feedback cycle is so slow for personnel changes, trying to defer it and “fix it for the next time” will never actually fix the issues. Internal hiring will force an organization to confront its issues, which can include:

  • Stifling managers that do not or cannot groom their reports for seniority and leadership.
  • A dysfunctional project that people do not want to work on and would be under-staffed if people were allowed to transfer.
  • Projects that depend on a couple of people, making them unable to transfer.
  • A general lack of learning and growth, perhaps because everyone is 100% allocated, with no slack time.
  • Work that is not challenging or evolving, causing the same experience over and over.
  • Valuing efficiency and specialties of individuals over utility and value.

All of these issues (and more) cause issues with internal hiring, but also are bad for the organization overall. Wouldn’t it be great if you could both fill a key role and address issues?

Is the risk too great of promoting a bad candidate? I don’t know: is the risk too great of hiring in an unknown quantity into a leadership position? Is the risk too great of having a candidate who wants a job change but your organization can’t give it to him or her?

If you are looking to do anything but shrink, you should always have ‘junior’ positions open and take the cream of the crop. This is especially true if you are outside a major tech hub.

There’s also another type of problematic hiring: adding resources to failing projects (whether outside or inside hires). We all know Brookes’ Law, that “adding people to a late project makes it later” but I can’t count how many times people do it anyway. If there are problems with a project, adding people is the worst way to address issues. “We need more resources” is a tantalizingly simple explanation for why something isn’t getting done, but I’ve never seen it be the actual reason. It is, like hiring leadership, a great way to disguise and distract from the real problems. This topic requires a separate post, though.

I also want to point out a perversion of ‘internal hiring’: creating an excess of managers and handing out seniority titles as candy. What I’m advocating here is when you need a manager, look internally, not to turn someone into a manager because they want it. Likewise, I’m not saying you should give someone a more senior title because otherwise you’d open up a senior developer spot, I’m suggesting you give them the responsibilities (say, team leadership) and see how they handle it.

It is much easier to hide the lack of internal hiring in technology companies because it is growing so quickly (there’s a need for external hires, and people can get jobs elsewhere if they become frustrated). But ultimately I see a dependence on external hires on the other side of the ‘firing as a tool’ coin. I don’t think you can do one without the other. They are inseparable from not just a cultural level but a practical one as well. It is about investing in your employees over looking for easy answers.

No Comments

“Delighting customers” is Lean’s secret handshake

by Rob Galanakis on 13/12/2013

Whenever I see the words “delighting customers” (which is, let’s face it, an awkward phrase) in a non-Lean context like a job description, I can feel the author winking at me. It tells me “we try to be Lean and if you get our drift you probably want to join us.” It instantly gives said company plenty of extra points.

I just wonder how long until “delighting customers” becomes a played out catchphrase (if it’s not already)?

No Comments

All languages need first class tuples

by Rob Galanakis on 6/12/2013

I was doing some work on a Flask app today and built up some chart data in Python that had a list of two-item tuples like [(<iso datetime string>, <value>), ...]. I needed to iterate over this same structure in JavaScript and of course was reminded how great it is that I can unpack these things easily in Python ala “for datestr, value in chartdata” because I was missing it so much in JavaScript. But rather than keep these as tuples and have to play around with non-idiomatic JS, I just made my tuple into a dictionary, so my chart data became a list of dicts instead of a list of tuples.

I really dislike having to use objects (or names, more precisely) for these micro data structures. Over time I’ve moved more and more away from creating classes for data structures in Python, a ‘best practice’ habit I brought over from C#. It is silly in a dynamically typed language. There’s nothing more clear about:

for entry in chartdata:
    chart.add(entry.x, entry.y)


for x, y in chartdata:
    chart.add(x, y)

In fact it’s probably less clear, because there is the totally unhelpful variable name “entry.”

At some point- maybe even three items?- it becomes more clear and self-documenting to use names (that is, dicts instead of tuples), but for the very common cases of simple collections used by nearby code, tuples and automatic unpacking can’t be beat!


Removing firing as a management tool (Pt 2 of 3)

by Rob Galanakis on 4/12/2013

So in my last post, I wrote about the possibility of taking hiring and firing off the table as a management tool. In this post, I will focus on the firing. Firing itself has two halves: individual dismissal as a way to fix performance problems, or layoffs as a way to fix organizational problems.

Lifetime employment has been a staple practice in Japan for decades. You can read more about it here, but it is as it sounds. Employees are only removed in extreme circumstances. Layoffs are almost unheard of. Tied up in this talk of firings is the modern management malpractice known as performance reviews. Enough has been written about how they are the greatest scourge ever inflicted on modern industry so I won’t go into it.

Using firing as a way to manage performance problems cannot work. The “10x Programmer” myth speculates that the difference between best and worst programmers is 10 times productivity. This sounds a lot but it puts every programmer within a single order of magnitude. That’s not that huge. Most of us are going to be clustered in the middle. So, all other things being equal, if you could make sure you are only hiring better people than you’re firing, you’re looking at a still small boost to productivity. And I want to be clear, the idea that you can hire a “better” person is unlikely- have you figured out how to systematically make your hiring better and proved it?

More likely, you are firing someone because their skillset or attitude is not so useful or relevant anymore. The QA person who won’t script. The programmer who won’t write unit tests. The designer who wants to work on something else. Firing an employee to fix this is ridiculous. A poorly performing employee is a management failure, not an employee failure. If the employee could have done better, she would have. By pushing the responsibility onto the employee, you are missing the opportunity to fix a problem. When you have someone in this situation, you sit down with other management and figure out why this situation exists and what can be done about it. And you sit down with the employee and engage with them about what’s expected and you come up with a plan. And if the plan isn’t met, you revisit it and see why it failed. You don’t fire the person. If they are truly unwilling to change, they will quit. Don’t worry so much about the “lost time.” You are already pulling along years of dead weight in your “high performing organization.”

There’s also the issue of incompetent management. Performance problems aren’t unique to individual contributors (ICs). Managers must be able to be “demoted” or reassigned. It should not be suicide to one’s career to move into another role that is a better fit (due to other circumstances or do to the manager’s abilities). A good idea is to not pay managers more. Title changes would hurt a lot less if they didn’t involve salary, and it wasn’t seen as the company punishing a manager but seeking to make a long term investment instead. It takes absolutely forever to fire an incompetent manager, and oftentimes it never happens. And clearly a poorly performing manager is probably an even bigger risk than a poorly performing IC. There need to be effective mechanisms to improve performance of managers even more than ICs!

Removing firing as a tool requires you to deal with performance problems in a way that teaches you something. It takes a coward to fire someone. It takes a leader to accept responsibility for a bad situation and work with involved parties to make it better. Odds are your employees want to make things better. If they really hate you and don’t give a shit, you are probably not the type of person to read this blog.

And then there are layoffs. Ugh. The inevitable outcome of some new executive, or a merger, or strategic change, or restructuring, or whatever. A layoff should be the absolute last option. It should come after executive pay cuts. It should come after a vote on temporary across the board salary cuts. It should come after stopping the acquisition of new companies. It should come after cutting back on spend-heavy areas like marketing. It should come with a mea culpa from executives, not a “we are rightsizing.” It should come with a deep and transparent retrospective of how things got out of hand (more than just “global financial crisis” please!). It should come with a plan on how to make sure this doesn’t happen again (and a commitment to never do it again). It should come with an understanding of how god awful and unnecessary layoffs are (Japanese companies are the obvious example but there are plenty of long-lived companies in Western industry that have never had layoffs). Of all the management cancers the technology industry has been able to avoid, layoffs are not among them.

Layoffs are the most blunt tool to ‘fix’ a problem, no matter what that problem is. They cause an untold amount of harm. The blunt trauma is almost impossible to learn from- you don’t get to keep the limb you’re chopping off to figure out what went wrong and what to do better (other parts of the body may come up with explanations but they’re probably wrong).

The executive that has layoffs is being as lazy as the manager that fires. Just because both of these things are emotionally difficult all around does not make them the correct thing to do. Avoid them not because they’re hard, avoid them because there are better ways of doing things. Fix your problems by actually fixing your problems. Take firings as a tool off the table completely.

Next up is hiring senior people from the outside.


Removing hiring and firing as a tool (Pt 1 of 3)

by Rob Galanakis on 2/12/2013

Inside of Mike Bland’s response to my Project as a Ship post is this comment of how he organized his Google Grouplet:

I had no power to hire or, truthfully, even fire anyone. The best I could do was persuade whoever was interested to participate, to give them the clearest objectives and parameters I could… and then do what I could to remove obstacles and make their jobs as easy and effective as possible.

Which got me thinking- should hiring and firing be totally restricted as a tool for addressing problems?

In Japan, lifetime employment at larger companies has been a staple for decades. And in spite of the inability to deal with problems by firing individuals, having layoffs, or hiring overseas executives, Japan managed to achieve world-leading levels of quality. They did this with very many constraints post-WWII and flagship firms have continued to consistently innovate and improve.

Hiring and firing is too often used as a panacea for whatever ails a company. It makes sense, right? If you have poor performers, getting rid of them will improve everything else. If you have a need for senior talent or leadership, you can just hire in from the outside if you don’t have an obvious internal candidate. Unfortunately, as decades of precedent show, neither of these ideas work very well. Of course, ‘well’ is not universal. I’ll define ‘well’ as a company who can make a sustainable profit (and, if desired, growth), supply good jobs, and improve the communities they are a part of. If your definition of ‘well’ is only to maximize the investment of shareholders, maybe these posts aren’t for you.

What would happen if we removed hiring and firing as tools managers could use to fix problems? I’ll break it down over the next two posts.

PS: A couple things. First, most of these ideas are inspired by W. Edwards Deming, so if these are things you are interested in I’d highly encourage you to read up on him, and Lean books about Toyota (especially Liker’s series). Second, I am not a big reference-giver unfortunately, but I have researched most of my claims, but if something is inaccurate or you disagree I’d be happy to discuss. Finally, I also want to make clear I’m not prescribing how to govern a country or run an economy. I’m talking about individual companies, and to some degree projects.

1 Comment

The Flawed Project-as-a-Ship Analogy

by Rob Galanakis on 30/11/2013

It is common for a project leader to see his or herself as the captain of a ship. The project is seen as the ship, and the developers are the crew. When storms hit- the project is behind schedule, over-budget, below quality, falling short of projections and expectations- the captain doubles down and starts to behave more captain-like to stabilize the ship.

There are many bad practices software development captains use to navigate stormy seas. Perhaps the worst of them is to silence dissent and remove “poor performers.” The idea is that by silencing dissenters and removing poor performers, you create smoother sailing for the ship overall, to help ride out the storm.

Unfortunately the ‘project as ship’ analogy is fundamentally flawed. The project is not the ship. The people are the ship. When you remove dissenters, you are ripping out vital structures. You are destroying your sails and your rudders. When you remove “poor performers”, you are throwing out vital reserves that you should instead be better utilizing. You are ripping holes in your hull. When you ‘cut back’ on investing in your people, you are foregoing essential maintenance and leaving your ship in disrepair. The project itself- the artifacts people create- has no place in the ship analogy. The project is a result of an equation of people, not a factor in it.

Focusing on stabilizing the project in times of trouble, rather than growing and nurturing the people that create it, doesn’t make for smoother sailing. It makes for a sinking ship.

1 Comment

Spotify’s DNA of Autonomy and Collaboration

by Rob Galanakis on 27/11/2013

Anders Ivarsson, an Agile Coach from Spotify, was kind enough to present his talk from Agile Ísland at our company on Friday morning. It was ostensibly about “scaling agile” but really it was a talk about how they’ve created a culture that prizes autonomy and collaboration.

I’m personally tired of hearing about how new, growing, successful businesses do things. In most cases, these evangelists really have no idea, and all you get is a post-mortem about choices a company made and tools they use, wrapped up in a presentation that these choices are integral to the company’s success and you should listen. In reality, most of the choices are made for no real reason, it’s just the way things turned out, and who knows if the company is successful because of these choices or in spite of them. And of course we don’t know if these choices led to an actual profitable or sustainable business or how they adapted to times when money wasn’t pouring in.

I felt different about this Spotify presentation even though it fit the mold. The reason is because Anders spoke very little about agile tools and very much about how the company is structured around autonomy and collaboration. Autonomy and collaboration are built into organizational structure. It seems monumentally difficult to create a command-and-control environment at Spotify. You’d have to restructure the entire company, top to bottom. Which means that a few bad apples (any company doubling its size every year has more than a few bad apples) won’t ruin the bushel. It would take a concerted effort of senior management to change that DNA, not a few incompetent bad hires.

This is the total opposite to most agile success stories you probably know of, where success is limited in area but especially depth and time, because most people are working in a command and control structure (even if the company doesn’t necessarily operate in a command and control fashion). Without structural reform, you can’t really do more than scratch the surface of truly agile or lean changes. And without structural reform, when a bad apple comes along into a ‘command and control but currently agile’ situation, all progress can be wiped away and it is incredibly easy to slip, almost immediately and without anyone observing, into a command and control system.

No Comments

A use for killing tasklets

by Rob Galanakis on 24/11/2013

A few weeks ago, I posted about Killing Tasklets, a feature of Stackless Python that allows you to abort a tasklet at any point. And it turns out I had a perfect use case for them just last week and things went swimmingly well.

We have a client program that controls the state of 3D objects, and a server program that does the rendering. The client calculates a serialized version of the server’s state (based on the client’s state) and sends it to the server. It does this through a publish/subscribe. The server receives the state and applies it to the current scene, moving objects and the like (of course we have other mechanisms for rebuilding the entire scene, this is just for ‘updating’ the attributes of the current object graph).

This causes a problem when the server takes longer to apply the new state to its scene than it does for the client to calculate it (maybe the client is super fast because it is caching everything). The server lags further and further behind the client. So when the server receives the ‘update’ command, it kicks off and stores the tasklet to do the updating. If another ‘update’ comes in while the previous update’s tasklet is still alive, it kills that tasklet and starts a new one. This way we get as smooth an updating as possible (dropping updates would cause more choppiness). This does require that updates are ‘absolute’ and not relative to other updates, and can be aborted without corrupting the scene.

Killing tasklets turned this into very straightforward code. In fact none of it other than the few lines that handle subscriptions on the server know anything about it at all. This sort of “don’t think about it too much, it just works like you’d expect” promise of tasklet killing is exactly why I like it and exactly what was fulfilled in my use case.

No Comments

Balancing Team Autonomy and Centralized Control

by Rob Galanakis on 20/11/2013

What is a leader on a multi-team Agile project supposed to do when there’s an expectation to make decisions that would bind all teams and for which he or she cannot hope to consider all viewpoints?

I often found myself in this position with generally two types of decisions to make.

The first were technological ones. These were easy because ultimately the work directly impacts a specific group the most. You just find that group, who of course are doing the work and understand the problem best, and just support their decision. It is well known that the people closer to doing the work have better information to make the decision. If you don’t trust them to make technological decisions, you are pretty screwed and should work on that. I was fortunate to have a very talented group of engineers that I fully trusted.

The second type were usually development decisions that did not have clear answers. Coding style, review process, Agile processes, etc. Decisions where everyone is a stakeholder. In these cases, I would always discuss the topic in an open forum, such as the relevant Community of Practice, or even via email. I would then take the decision of the group and make it “official policy.”

It’s important that these discussions not discipline discussions, or leadership discussions; they must be open discussion in a zero-barrier-to-entry forum. It is like early RFCs: if people are interested, they can discuss. If they are not interested, they do not have to take part, but they will have to abide by it. It was also my policy to revisit decisions every few  months (maybe 6 for project-wide changes, or 2 for changes scoped to a team or two). This way people would have less fear of change. If something doesn’t work out, we can always go back to how it was. Where decisions were made on a discipline level, or the expectation to revisit these types of decisions wasn’t made clear, I think development decisions did not work well.

I feel this style of decision making walked a good balance between:

  1. Actually making decisions. It’s too easy for Agile project leadership to hide behind “team autonomy” as an excuse for not making decisions (technological, developmental, or otherwise) or being held accountable.
  2. Respecting team autonomy. Ideas could be piloted on not-unwilling teams with the understanding that things could go back to how they were if they don’t like it.
  3. Consistency between teams. We could force changes across teams, creating consistency where beneficial, without feeling so bad, because everyone who was interested could have a say.

This is what I aspired to, at least. And when I followed it, decisions tended to be positive. When I lost sight of it (or before I gained clear sight of it), my decisions were pretty average.

No Comments