Archive of published articles on August, 2013

Back home

The perils of bunch/bundle?

25/08/2013

I had some discussion on Friday about static vs. dynamic typing (grass is always green, eh?), and our frequent use of the “dictionary with attribute-like notation” object came up (commonly called a “bunch” or “bundle”). A number of people were saying how bad/unsafe it is. Not convinced, I brought the source code up on the projector and asked people: “If I were to set the ‘__’ key, what would happen?” My thinking was that, if this is a data container like a dictionary, what happens when the data conflicts with non-data keys?

A room full of programmers, and no one could answer (actually I still don’t know, nor do I remember if it raised a KeyError and AttributeError in the appropriate circumstances, but I’d guess not). I pointed out the reason this type of object is a problem is because, even though we have several thousand uses of it and it’s been around for a decade, how it works isn’t clear or documented. Static typing doesn’t change that. Bad code is bad code and there’s heaps of bad code in wonderful statically typed languages.

It so happens we have several more implementations of the same object. The only one that has documented edge case behavior is in probably the highest quality area of the codebase. Incidentally, it is only used during testing. So maybe bunches aren’t that useful after all…

1 Comment

Twitter’s reorg takeaways

19/08/2013

I just read through a great Twitter blog, New Tweets per second record, and how!┬áIt covers in some depth the changes to their engineering organization over the last three or so years. As CCP is undergoing similar technical stresses (we hit our PCU record recently), and responding with similar actions, I thought I’d write down my takeaways and personal feelings.

Interpreted languages are fast enough or too slow. Ruby helped Twitter get where it is, and without such insane peak performance needs I’m sure they would have stayed in Ruby. We’re actually about to embark down rewriting some core systems from Python into C++, because we can’t always throw (more|better) hardware at the problem.

Architect your organization the way your want your software to be architected. This is derived from Conway’s law (like, “4 teams working on a compiler will develop a 4 pass compiler”), and I feel is central. If you want to rearchitect your technology, you need to reform your organization. There’s no way around it. The basis of discussion for those organizational changes should be how the technology should look.

Prefer interfaces at the service level. This isn’t always possible (core libraries or frameworks, fat clients, etc.), but should be preferred. It is a natural boundary. Though, interfaces at the module/package/class level can work fine for a certain scale (up to quite large, I imagine). Twitter has more open Software Engineer positions than most projects have programmers! But SOA is a great thing, for technology and for organizations.

Self-organized teams around services are effective. Self organizing teams have been one of the keys to Agile’s effectiveness (even when Agile development principles aren’t followed). It is difficult to scale Agile up to multiple teams, though, so dividing at service boundaries is a convenient way to reduce how large Agile must scale up. (As an aside, I don’t mean to say “Agile doesn’t scale” or that other development methodologies scale better, it’s just really difficult to scale up software development)

Monolithic DBs will be a bottleneck. Twitter came up with some clever solutions for sharding and balancing that worked for Twitter. If you have a monolithic DB under great strain, you will need to get creative with your own solution.

No Comments