Twitter’s reorg takeaways

by Rob Galanakis on 19/08/2013

I just read through a great Twitter blog, New Tweets per second record, and how! It covers in some depth the changes to their engineering organization over the last three or so years. As CCP is undergoing similar technical stresses (we hit our PCU record recently), and responding with similar actions, I thought I’d write down my takeaways and personal feelings.

Interpreted languages are fast enough or too slow. Ruby helped Twitter get where it is, and without such insane peak performance needs I’m sure they would have stayed in Ruby. We’re actually about to embark down rewriting some core systems from Python into C++, because we can’t always throw (more|better) hardware at the problem.

Architect your organization the way your want your software to be architected. This is derived from Conway’s law (like, “4 teams working on a compiler will develop a 4 pass compiler”), and I feel is central. If you want to rearchitect your technology, you need to reform your organization. There’s no way around it. The basis of discussion for those organizational changes should be how the technology should look.

Prefer interfaces at the service level. This isn’t always possible (core libraries or frameworks, fat clients, etc.), but should be preferred. It is a natural boundary. Though, interfaces at the module/package/class level can work fine for a certain scale (up to quite large, I imagine). Twitter has more open Software Engineer positions than most projects have programmers! But SOA is a great thing, for technology and for organizations.

Self-organized teams around services are effective. Self organizing teams have been one of the keys to Agile’s effectiveness (even when Agile development principles aren’t followed). It is difficult to scale Agile up to multiple teams, though, so dividing at service boundaries is a convenient way to reduce how large Agile must scale up. (As an aside, I don’t mean to say “Agile doesn’t scale” or that other development methodologies scale better, it’s just really difficult to scale up software development)

Monolithic DBs will be a bottleneck. Twitter came up with some clever solutions for sharding and balancing that worked for Twitter. If you have a monolithic DB under great strain, you will need to get creative with your own solution.

No Comments

Spreading TDD throughout your company

by Rob Galanakis on 4/07/2013

First, the pitch- CCP is looking to hire someone to come to Iceland and teach TDD with Python for a couple days in August. We’re looking for someone that’s taught TDD before. We already have people who have been doing TDD for a while now and to support it in the long run, but we want to get more people on board and grow some recent evangelists. If you’re interested, contact me or rgalanakis@ccpgames.

Now to make this post not void of content, I just want to give a little bit of history, and insight I’ve gained while pushing TDD at CCP for the last two years.

I joined CCP two years ago and since before day 1 I’ve been doing TDD. Since then, I can’t say it’s caught on like wildfire (we have a legacy Python codebase of around 2 million lines), but we’re at a point now where it’s taken hold in enough areas, and proved itself both doable and useful, that we’re ready to start ‘pushing’ it on engineers, rather than just waiting for individuals to ‘pull’ it.

I started by focusing on a non-critical area of the code (internal tools) and wholesale replacing functionality rather than trying to retrofit legacy code because this was non-critical code I could generally afford to not replace all of its functionality, or have some things “broken”. A few months later, a different team working on a new area of the code embraced unit testing and TDD, so we had it working from two angles.

Eventually we needed to test more than pure Python. We needed to test code that’d only run in our game environment, or Autodesk Maya. So I developed some systems to run code in a remote environment from a host Python process, so we could still run our tests from PyCharm or whatever pure Python process, but under the hood it’d bootstrap one of these other processes, tell it to run the test code, and report the result to the original Python process.

This was a big ‘aha!’ moment that made TDD go from impractical to imminently possible. We conquered the biggest hurdle to automated tested we had. It was now possible to find and grow TDD evangelists working in game code, and start learning there. It is a much more difficult problem to test! But something more interesting happened. This helped crystallize a movement where we wouldn’t need this craziness. Unit tests started to become too important, so we did things like:

  • Build versions of our binaries for various vanilla Python flavors, not just our proprietary executable environment.
  • Created a new ‘home’ under which we put cohesive Python packages and expect good code (it’s hard to develop well in the face of 2 million lines of legacy).
  • Extracted code out of our ‘monolith’ and these packages (or totally separate environments). Our ‘monolith’ actually used a proprietary importing mechanism which made it impossible to use outside of our game environment, and depending on all sorts of builtins.
  • Adopt better tools. Since we started writing vanilla code, and using standard testing tools, IDEs like PyCharm had real benefit over Sublime or TextPad.
  • We removed our proprietary importing mechanism and are removing as many things as possible that diverge our codebase from vanilla Python.

Certainly I’m not saying I or TDD were the only drivers of these improvements, and to a large degree our testing push was in the right place at the right time. And certainly it’s not been a short time since we started. But now we’re ready to push out TDD more aggressively, with official approval. Which is why we’re looking for a trainer to come in and help educate about 15 Python devs for a day or two.

The insight I’ve learned is that the common-sense tips people give about creating change are generally correct. Start with small wins, find early adopters and make sure they are happy, keep pushing steadily, overcome harder problems, change peoples’ minds with results, etc.


Python Enrichment and PitP

by Rob Galanakis on 18/01/2013

When I was starting my job at CCP, I posted about some things I wanted to do as a lead. I’ve been through two releases with the Tech Art Group in Iceland (and for the past 6 months or so been the Tech Art Director here) and figured I’d measure my performance against my expectations.

Training sessions: I’m proud to say this is a definite success. I was the initial Coordinator for our Reykjavik Python Community of Practice (studio-wide volunteer group that discusses python in general and its use at CCP), where I started two initiatives I’m very proud of. One is the weekly ‘enrichment session’ where we watch a PyCon video, go through a demonstration or tutorial, etc. These have gone over great, the trick is to do them even if only 4 people show up :) The other is Python in the Pisser, a python version of Google’s Testing in the Toilet. I hope we can open source the newsletters and maybe share content with the outside world. More information on that coming in the future.

Collective vision/tasking: We run an XP-style team on tech art so I like to think my TAs feel ownership of what they are working on. In reality, that ownership and vision increases with seniority. We are actively improving this by doing more pairing and having more code reviews- the team is at a level now where they can review each other’s work and it isn’t all up to me to mentor/train them.

Evolving standards and practices: We started in Hansoft, moved to Outlook Tasks, and settled on Trello. We’ve discovered our own comfortable conventions and can discuss and evolve them without getting into arguments or ‘pulling rank’.

Community engagement: The CoP and mentoring has definitely done some work here. I try and give everyone 10%/10% time, where 10% is for ‘mandatory’ training or non-sprint tasks, and 10% is for their personal enrichment.

Move people around: We haven’t had much of an opportunity for this but we did change desk positions recently :) The art team is too small to have many opportunities and all graphics development is on one scrum team.

Code katas: We had one and it was mildly successful. We plan to do more but scheduling has been difficult- we do two releases a year, and DUST introduces complications in the middle of those releases, but we’ll be doing more things like it for sure.

I’ve also been doing very regular 1-on-1′s and, I hope, been getting honest feedback. Overall I am happy with my performance but can improve in all areas, even if that means doing more of the same (and becoming a better XP team).

Anyone want to share what successful cultural practices they have on their team/at their studio?


Maya and PyCharm

by Rob Galanakis on 15/01/2013

Recently at work I had the opportunity of spending a day to get Maya and PyCharm working nicely. It took some tweaking but we’re at a point now where we’re totally happy with our Maya/Python integration.

Remote Debugger: Just follow the instructions here: We ended up making a Maya menu item that will add the PyCharm helpers to the syspath and start the pydev debugger client (basically, the only in-Maya instructions in that link).

Autocomplete: To get autocomplete working, we just copied the stubs from “C:\Program Files\Autodesk\Maya2013\devkit\other\pymel\extras\completion\py” and put them into all of the subfolders under: “C:\Users\rgalanakis\.PyCharm20\system\python_stubs” (obviously, change your paths to what you need). We don’t use the mayapy.exe because we do mostly pure-python development (more on that below), and we can’t add the stubs folder to the syspath (see below), so this works well enough, even if it’s a little hacky.

Remote Commands: We didn’t set up the ability to control Maya from PyCharm, we prefer not to develop that way. Setting that up (commandPort stuff, PMIP PyCharm plugin, etc.)  was not worth it for us.

Copy/Paste: To get copy/paste from PyCharm into Maya working, we followed the advice here: Note, we made a minor change to the code in the last post, because we found screenshots weren’t working at all when Maya was running. We early-out the function if “oldMimeData.hasImage()”.

And that was it. We’re very, very happy with the result, and now we’re running PyCharm only (some of us were using WingIDE or Eclipse or other stuff to remote-debug Maya or work with autocomplete).


A note about the way we work- we try to make as much code as possible work standalone (pure python), which means very little of our codebase is in Maya. Furthermore we do TDD, and we extend that to Maya, by having a base TestCase class that sends commands to Maya under the hood (so the tests run in Maya and report their results back to the pure-python host process). So we can do TDD in PyCharm while writing Maya code. It is confusing to describe but trivial to use (inherit from MayaTestCase, write code that executes in Maya, otherwise treat it exactly like regular python). One day I’ll write up how it works in more depth.

1 Comment

Best practices for temp files

by Rob Galanakis on 13/01/2013

It seems the more middleware we use, the more we need to work with temporary files. So for Tech Artists, we usually work with temp files a lot! So it’s important we follow best practices when we need to work with temporary files.

Note: I’ll be using python functions and terminology but this applies to any language.

It goes without saying but, use the tempfile module. Don’t use environment variables or roll something yourself (I’ve seen both done).

Never hard-code the location of a temporary file. I’ve used middleware that does an equivalent of “os.path.join(gettempdir(), ‘’)“ to get some path. Which means running multiple processes at the same time causes file lock errors! I’ve had to change the temp directory before calling these processes.

Either your directory or your filename should always be programmatically looked up: so “mkstemp(dir=os.path.join(gettempdir(), ‘myscratchfiles’)“ is acceptable, as is “os.path.join(mkdtemp(), ‘’)“.

Try and clean up your files and folders! I say ‘try’ because it isn’t always possible. Sometimes your library may need to return the path to some file it generates. In that case, rather than rely on the caller to clean up the file (which would be a bizarre dependency and docstring!), you’re better off to take in the ‘output path’ to your library, and write the result file there. And your library can clean up the intermediate temp files. That way the caller can be responsible for deleting the result file if it needs (hopefully it doesn’t) because it knows more about the path.

In a lot of cases, I don’t bother cleaning up temp files for internal software. Management of temp files takes a non-negligible amount of work, and if I can keep the software simpler by not worrying about it, I will. I’d rather rely on the developers to keep their hard drives in good shape, which they’ll do anyway.

If you’re running something that will generate a lot of temp files (maybe your test running script), you can also set your temporary directory to a temporary subdir, and then clean that up. Something like:

oldtemp = tempfile.gettempdir()
newtemp = tempfile.tempdir = tempfile.mkdtemp() # Maybe set os.environs TMPDIR/TMP/TEMP as well?
try: # Or whatever processing you need to do
    tempfile.tempdir = oldtemp

And lastly (it’s last because I’m sure people have some religious objection to it), I’ve actually created a custom mktemp function that just calls mkstemp and closes the file descriptor. Just getting a path is useful and worth the performance overhead in (IMO) the vast majority of cases, especially since we often just need the filename to pass to some external process.

What advice do you have for working with temp files (and where is my advice poor)?


How wide are your interviews?

by Rob Galanakis on 20/12/2012

I’ve been a part of  and interviewed at companies where the interview process was not just long but also very wide- people from different teams and departments interview a candidate. (ie, a Microsoft-style interview process)

At my current company, our last two Art hires have had a much more abbreviated hiring process- the Art Manager, myself, and the Art Producer (and the Art Director if needed).

So I’ve been through and a part of interviews at both ends of the complexity spectrum. I’ve been thinking about it recently and I’ll share some thoughts (and I’d also love people’s own opinions and experiences).

  • Only a few people are generally good interviewers. If this doesn’t correspond with your Leads/Directors/Managers, there’s a problem. So I am not sure the depth of the interview varies greatly between the two styles.
  • On the other hand, interview experience is important for more junior members, and if they don’t get experience they’ll not turn into good interviewers.
  • Only a few people are generally responsible for making the decision anyway, and I’ve never actually seen a contentious decision: if one or two people have a thumbs down opinion, it’s enough of a red flag for everyone else. What’s the point of six or twelve people sitting around and pretending to decide?
  • How often are interviews really surprising? You know in the first couple minutes whether you’ll accept someone or not, or whether they’re in the middle (usually meaning you need to find a better candidate).

So given those experiences I’m going to try do the following for the interview process for Tech Artists:

  • See who wants to be part of the interview. Cultivating this skill is vital, but it isn’t for everyone, and people who have no interest shouldn’t waste their time.
  • Expand the interview to include those people- it should definitely extend past the manager/lead level.
  • Be rigorous, and don’t bother hiring anyone that isn’t unanimously liked. When the candidate pool is thin or work is slipping, it’s tempting to take a chance. It’s not worth the risk (in my opinion, of course).

I’m curious how other studios handle it, and what people think of those processes?

*I feel obligated to mention that if your studio is a management-run people-churner, please gripe at your managers and not me for writing this post!

No Comments

Development directors?

by Rob Galanakis on 18/12/2012

I’ve worked with (for?) several development directors. Some have been talented individuals, some incompetent. But invariably, the role has (in my experience) been useless. I’ve always seen it as a needless position on the producer/PM hierarchy. And oftentimes an add-on position due to explosive producer/PM headcount growth.

I understand what creative/technical/art/design/qa/(product|game) direction is. I have no idea what “development direction” is. And if I do, I have no idea how it’d be beneficial. I mean, the few decisions like “we are using Hansoft for scheduling and backlog management” can be made at the project producer level, and most other ‘development direction’ I can think of negatively infringes on team autonomy.

Has anyone actually had success with the role of development director? Not just a super-talented individual who happened to do a good job, but an actual use for the role?


Python Singletons

by Rob Galanakis on 18/11/2012

In my opinion, good python libraries and frameworks should spend effort guiding you towards the ‘pit of success’, rather than trying to keep you from failing. They do this by spending most effort on things related to the critical path- clear interfaces, simple implementations, thorough documentation.

Which is why singletons are, to me, the worst form of framework masturbation in python. You will never be able to stop people from doing something stupid if they’re determined (in pure python). In the case of a singleton, that means instantiating more than one instance of a type. So spending effort on ‘designing’ singletons is not just a waste of effort, but actively harmful. Just provide a clear way to use a single instance, and your system should fail clearly if it detects an actual problem due to multiple instances (as opposed to, trying to detect multiple instances to keep said problem from happening).

The best method for singletons in python, then, is- whatever is simplest!

  1. Some form of module or class state is, to me, the clearest. It requires someone reading or using your code to know nothing more than the most basic python. Just prefix your class def with an underscore, and expose an accessor function to an instance stored on the module (or on the class). The capacity for failure is minimal and the behavior is clear (it requires no behavior modification to the type itself).
  2. Overriding __new__ is pretty bad but OK. It requires someone to understand the subtleties of __new__, which is a useful thing to teach someone but, are singletons really the time and place?
  3. Using a metaclass is a terrible solution. It has a higher likelihood of failure (how many people understand the nuances of metaclasses!?). Misdirection even for people just reading your code, trying to understand your type’s behavior. Avoid.
The question to ask yourself before doing any of this is, “is a singleton a technical requirement or an architectural preference?” Ie, a single instance of an application event loop (QApplication, etc) I’d consider a technical requirement and make it foolproof (in C?). But technical requirements are few and far between and should be driven by underlying system/OS requirements rather than your code’s design or architecture. If it’s an architectural preference- “there should only be one instance of this manager/window/cache”- there’s absolutely no reason to confuse your code (especially you object’s behavior!) to achieve it. Just use design, documentation, and examples, to show people the right way to use it.

Taking your dog to Iceland

by Rob Galanakis on 17/11/2012

In late 2011, my wife and I imported our Boston Terrier, Shoni, to Iceland. Since importing pets commonly comes up on foreigner discussions, I thought I’d dedicate a post to our experiences (sorry, no tech writing today!). I’ll also mention, this is specifically for dogs, at the time we did it. Cats may be different (easier and more people have done it, from what I hear), and rules may change. I’m also not going to bother linking to forms- this is a personal experience, not a guide, and there is some amount of work involved in moving your pet to Iceland!

Overall, the process isn’t too hard. Information is pretty clear (and in English), everyone was very accommodating, and there were no surprises from Iceland’s side. I wouldn’t bother getting a service to do it, I’d just do it myself. And our dog came out no worse for the wear.

Anyway, on to the timeline (I’ll list prices at the end):

1. In early September, I applied for a permit from MAST, which is good for up to a year. I should have done this much earlier. I needed to get the permit at least 30 days (something like that) before importing Shoni, which meant by the time I got the permit, she couldn’t come with us in October. This threw everything off and caused a lot of stress and extra shots. So get your permit ASAP. The rest of the process is filled with timing restrictions, don’t mess this one up.

2. There are two quarantine places. Once is in the far north (Hrisey, near Akueyri), one is nearer to Keflavik. They stagger their intake, so one takes pets for a few days in the middle of the month, the other at the end/start of the month. If you have a choice (and are living in the capital area), I’d choose the one near Keflavik, to avoid an extra return flight and less transport overall. I’ve also heard their English is better. Shoni ended up going to Hrisey though, because I screwed up the permit and that’s what worked out timing-wise.

3. There are certain shots that the dog needs to have at least 60 days from departure. There are other shots that she needs no more than 30 or 60 days from departure. Read everything and schedule everything in advance. Don’t mess this up! Though we did a little bit but talked to MAST and everything was fine.

4. Make sure you have an airline-ready travel crate, and make sure your dog is used to sleeping in it! A regular crate won’t do, you may need a special crate for air travel. We lucked out, Casady’s aunt and uncle had one they weren’t using, from their recent international move.

5. Shoni flew from Texas to New York in mid-October, after our wedding, in coach with my Aunts. This was the least harrowing part of the entire trip, even though a 10kg dog in the cabin is too big! If you can book your pet in the cabin, do it (it’ll also be cheaper). Try it even if they may be too big- she flew Jet Blue, and it made me like that airline even more. If she must go cargo, I’ll note that airlines have restrictions and cautions for certain dog breeds during certain months. Boston Terriers are brachiocyphallic (snub-nosed), so they can overheat and die during travel, especially during summer months (usually sitting out on the tarmac). Different airlines have different records and different breeds have different accident rates. But do your research and be safe (if the dog is worth the price you’re going to pay to move her, she’s worth this risk, I’d say). We sent Shoni’s (empty) crate as baggage.

6. It was difficult and a huge burden to place on someone (my Aunts) to prep a dog for an international move. We did a bunch of stuff in Austin, but there was stuff that needed to be done 30/10 days before departure, that my Aunts needed to do. Shoni ended up getting several unnecessary shots; there was lots of confusion and stress in the month she was with my Aunts. On the other hand, breaking the trip into two parts was easier on Shoni. I would have been scared for her life if she went on an 18 hour trip in her crate. The key takeaway is that better planning and less procrastination would have made things easier, but even with mistakes along the way everything turned out fine.

7. Again, read the paperwork. There’s stuff you need to fax 5 days before your pet arrives. There was, not surprisingly, a lot of stress in the last few days. Lots of calls with MAST and my family and making sure everything was good to go.

8. With everything set, Shoni flew Icelandair cargo in mid-November from JFK to Keflavik. It was painless and we got confirmation she arrived, the worst parts were over. The quarantine people picked her (and presumably other pets) up, drives them to Hrisey (or wherever your quarantine is), and they start their quarantine. There is no way to interact with your pets AFAIK, even if you are on the same flight (sometimes you may see them in the luggage area but it isn’t like you can take them out to pee!).

9. Once in quarantine, we asked for updates/photos, and got them. The “warden’s” English was not great so it was difficult to communicate, so I needed to have a friend translate some emails between English and Icelandic. Shoni looked nervous and the place was obviously pretty sterile, but she looked safe and healthy. Shoni is very well-adjusted and adaptable, so if you have a nervous pet she may fare worse. Also, I think you can visit your pets- but I wasn’t about to go to Akureyri as a new resident in December, but I probably would have visited her in Keflavik.

10. After 4 weeks (right before Christmas, for us!), Shoni’s time was up. And here’s another reason to choose Keflavik over Hrisey- Icelandic winter weather is fickle and her flight was delayed several times. She was on one of the last flights from Akureyri. We picked her up from the airport, took her home in a cab, and all was well. She didn’t have any of her toys or blankets she was sent with- the stuff has to be boiled and rarely survives it.

11. Costs. All are approximate.

  • Permit: $240 (including transfer fee)
  • Flight Austin->JFK: $100
  • Flight JFK->Keflavik: $450
  • Akureyri->Reykjavik: $100
  • Quarantine: $1600 (varies with pet size)
  • Customs fee: $100
  • Vet bills: $700 or so, not sure
  • Total: About $3500

In the end, it wasn’t a question of whether or not it was worth it- there was no question, she was coming. However, procrastination complicated things. And it is expensive. Bigger dogs will be more expensive, cats are smaller and require less shots so would be cheaper.

Anyway, I hope this post is helpful, and feel free to ask me any questions about my experience.


Is QA a good stepping stone?

by Rob Galanakis on 9/11/2012

I’ve always heard that it was difficult to move from QA into development (game design/programming/art/production). I thought this was smart- QA people should be there to be QA people, not doing a job only because they hope it would lead to something else.

And at some companies, it works. My previous job was at a well-regarded studio, working on a huge game and IP, in a wonderful city, and lots of developers were looking for work (and we had the money to hire them, if not keep them after ship…). People really wanted to work there (I once interviewed someone as I was near the end of my tenure there, and told him pretty clearly, “you won’t like programming here.” And of course he took the job anyway). It didn’t make sense to spend effort training QA people when you could just hire the people you wanted directly.

However, I now work at a place that is much more difficult to hire for. Outside people respect the company and know the game, but the audience is smaller than one of the biggest IPs and most successful game studios of all time. And especially, it’s in Iceland. Iceland is a wonderful country, but we’ve had several people turn down offers or on-site interviews because their family said “no way” (myself almost included).

In my opinion, it makes more sense to hire QA people (often out of the EVE or local Icelandic community), and set them up on a career path that leads outside of QA. There will never be a serious shortage of applications for a QA job, but it can take a year to hire and relocate a senior person (and you are taking a big risk on that time and expense!). And to a large extent, this is what we do (and while we have sometimes failed at providing career paths, I still think it remains the goal).

I quite like this new strategy- not only does it produce good results over time, it also seems to have more integrity- QA people are too often treated like second class citizens, rather than devs-to-be. I’m not sure if my mind will change if I move on to another studio, but it’s at least how I feel now. I wonder how other studios approach this issue?

(Side note- I am not saying there isn’t a need for senior and permanent QA people, I’m just saying, there are lots of people in QA that don’t want a career in QA. There should also be career development inside of QA, but that shouldn’t be the only career development)

No Comments

Switch to our mobile site