Website and Blog of Rob Galanakis: Tools Programmer, Technical Artist/Animator

When I was starting my job at CCP, I posted about some things I wanted to do as a lead. I’ve been through two releases with the Tech Art Group in Iceland (and for the past 6 months or so been the Tech Art Director here) and figured I’d measure my performance against my expectations.

Training sessions: I’m proud to say this is a definite success. I was the initial Coordinator for our Reykjavik Python Community of Practice (studio-wide volunteer group that discusses python in general and its use at CCP), where I started two initiatives I’m very proud of. One is the weekly ‘enrichment session’ where we watch a PyCon video, go through a demonstration or tutorial, etc. These have gone over great, the trick is to do them even if only 4 people show up :) The other is Python in the Pisser, a python version of Google’s Testing in the Toilet. I hope we can open source the newsletters and maybe share content with the outside world. More information on that coming in the future.

Collective vision/tasking: We run an XP-style team on tech art so I like to think my TAs feel ownership of what they are working on. In reality, that ownership and vision increases with seniority. We are actively improving this by doing more pairing and having more code reviews- the team is at a level now where they can review each other’s work and it isn’t all up to me to mentor/train them.

Evolving standards and practices: We started in Hansoft, moved to Outlook Tasks, and settled on Trello. We’ve discovered our own comfortable conventions and can discuss and evolve them without getting into arguments or ‘pulling rank’.

Community engagement: The CoP and mentoring has definitely done some work here. I try and give everyone 10%/10% time, where 10% is for ‘mandatory’ training or non-sprint tasks, and 10% is for their personal enrichment.

Move people around: We haven’t had much of an opportunity for this but we did change desk positions recently :) The art team is too small to have many opportunities and all graphics development is on one scrum team.

Code katas: We had one and it was mildly successful. We plan to do more but scheduling has been difficult- we do two releases a year, and DUST introduces complications in the middle of those releases, but we’ll be doing more things like it for sure.

I’ve also been doing very regular 1-on-1′s and, I hope, been getting honest feedback. Overall I am happy with my performance but can improve in all areas, even if that means doing more of the same (and becoming a better XP team).

Anyone want to share what successful cultural practices they have on their team/at their studio?

I’ve worked with (for?) several development directors. Some have been talented individuals, some incompetent. But invariably, the role has (in my experience) been useless. I’ve always seen it as a needless position on the producer/PM hierarchy. And oftentimes an add-on position due to explosive producer/PM headcount growth.

I understand what creative/technical/art/design/qa/(product|game) direction is. I have no idea what “development direction” is. And if I do, I have no idea how it’d be beneficial. I mean, the few decisions like “we are using Hansoft for scheduling and backlog management” can be made at the project producer level, and most other ‘development direction’ I can think of negatively infringes on team autonomy.

Has anyone actually had success with the role of development director? Not just a super-talented individual who happened to do a good job, but an actual use for the role?

There was a recent AltDevBlogADay post about Negative Developers and Team Stability that hit home. It’s not that I think the advice was particularly interesting (good, standard stuff), it’s that it reminded be that I’ve been a negative developer.

I don’t know what I could have done differently. I just wasn’t happy at work, and there was little I could do to change it. The quality of my work was apparently very good, I was just terrible for morale, because I was either 1) pissing people off or 2) encouraging people to be pissed off at the problems I/we saw. Eventually I got the best advice I’ve ever gotten (which deserves its own blog post), and left the company. I went to the right place and became a positive developer.

And that’s sort of what struck me about the article and about how we typically deal with negative developers. Some developers are just not a good fit, regardless of how amazing their work is. If someone is negative because she is “culturally incompatible”, because there’s nothing you or your manager can do to fix it. And it is worth it to have a frank discussion about whether that person can ever be happy without changes to the studio, and if that person says ‘no’, you should discuss plans to part with mutual respect at a mutually agreed date.

I had to put in my two weeks at my last job to have this advice given to me by the President (GM? Can’t remember) at the time. It convinced me to un-quit, and to stay on another year. It ended up being a miserable year in many ways, but it was the right thing to do and worked out for the best. As managers- and friends and team members of negative developers- we need to keep this advice in mind when dealing with negative developers (and ourselves).

Some people can find me a bit over-earnest in my quest for automation. I’ve finally figured out how to know whether something is worthwhile to automate.

Are you making any decisions when you do this?

And if someone is making decisions that may be unnecessary:

Can we get from A to B without making any decisions?

I value the people I work with and I want them to be more effective. Sometimes people are afraid I’ll automate them out of a job, or they like a manual part of their workflow. More often they take pride in being the manual bottleneck behind a tricky or brittle process.

Humans are valuable because we can make complex decisions very quickly. When I automate an area someone isn’t immediately comfortable with, it’s not because I don’t value what he or she does. On the contrary: because I value their skills and time, I want to see them doing something that requires their unique abilities.

A poster on tech-artists.org who is using Autodesk’s CAT asks:

 The problem I´m having: I can control the ears now manually, after setting up the reactions, but apparently the CAT system isn´t respecting any keyframes I set on the facial controls, neither in setup nor animation mode.

He already hinted at the answer in the same post:

Even though I´ve heard a couple of times not to rely on CAT in production…

So there’s your answer.

Never depend upon closed-source frameworks that aren’t ubiquitous and proven.

It’s true of CAT, it’s true of everything. And in fact it is one reason I’ll never go back to developing with .NET on Windows (the 3rd party ecosystem, not the framework itself). If you don’t have the source for something, you 1) will never fully understand it, and 2) never be able to sustain your use of it. When I was at BioWare Austin, and the Edmonton guys decided to switch to CAT, I was up in arms for exactly this reason. We had an aging framework- PuppetShop- but it worked well enough, we had the source, and acceptable levels of support (Kees, the author, is/was readily available). Instead, they switched to a closed-source product (CAT) from a vendor that has repeatedly showcased its awful support (Autodesk), and headache ensued. Fortunately I never had to deal with it and left some months after that. Without the source, you’re dependent upon someone whose job it is to provide support just good enough so you don’t leave their product (which is difficult since they own everything), and bad enough that you have to hire them to fix problems (they pride themselves on this level of support, which I call extortion).

As for the ubiquitous and proven part: unless this closed source solution is incredibly widespread (think Max, Maya, etc.), and has a lot of involved power users (ie, Biped is widespread but difficult to get good community support for because it attracts so many novices), it means you have to figure out every single workaround because you can’t actually fix the problem because you don’t have source. Imagine working with no Google- that’s what you give up when you use a closed-source framework that isn’t an industry standard.

So don’t do it. Don’t let others do it. If you’re currently doing it, think about getting the source and if you can’t do that, start making a backup plan.

In December, I made a blog post about being happy. A anonymous (of course) commenter said:

Hey Rob. 5 months? Are you really this shortsighted? It’s like watching a kid say he’ll NEVER get tired of THIS toy. Frankly, if you don’t get tired of it you stopped growing. Take it as a good learning experience for the time when you forge your own destiny, but if you aren’t back to being a grumpy asshole in 6 months I’ll eat my hat. Not that I wear hats.

So 6 months (and a released expansion) later, have things changed? No, not one bit. In fact, late-night-drunk-rants have growth even more positive all around. Few of the bitch-sessions I’ve been used to in my career. It is an unfamiliar feeling. But addictive.

So why am I not back to being a ‘grumpy asshole’? Because there’s nothing to be grumpy about. There’s plenty wrong, but we’re moving at a steady pace in the right direction. And everyone is on board that it’s the right direction. No “we’ve decided this is the right direction so STFU and do what you’re told”. This happens because the talented people that complain loudly have several constructive outlets (I put myself in that group, at the risk of sounding egotistical).

One of the most exciting are communities of practice. Read the Wikipedia article if you like, but really they are a company-sanctioned forum for people with axes to grind to argue. Then we figure out what we want to do, and how to do it. And then we do it on some 15% time for skunkworks projects we’re able to reserve. Being able to talk about our common problems, across teams and responsibilities- we have everyone from tech artists to server programmers to the CTO there- is a refreshing experience.

(If you’re at a large company with lots of programmers who all aren’t on the same team, I’d highly suggest forming your own communities of practice).

The people who get shit done and can convince you of the way forward are given the responsibility to get more shit done. It’s nice. I think it has to do with the open source and python mentality.

In the comments on post “Internal Tools Only Require the Critical Path“, Robert Kist points out a few problems to think about when developing internal tools that may have to be used by outsourcers.

I absolutely agree and writing outsource-compliant tools are something many TA’s (including myself) can struggle with. The biggest things are:

Security: Assume your users do not have administrator rights. This shouldn’t be a problem if you are doing things the “right way”, rather than the “hard-coded expedient at the moment” way.

SCM Clients: Do your outsourcers have SCM clients? If not, what does that mean if you scatter your SCM interaction throughout your tools? Are you observing best practices and trying to make your SCM interaction transactional, or are you just making calls at the earliest possible moment?

Network resources: Do your tools require access to network resources, and if so, do the outsourcers have access to them? This could be a database, or a file on the network. Consider how you can break or mock these dependencies (or get rid of them entirely).

Latency: If your tools do require network resources, are they hitting the database/network constantly? If so, expect your tools to be incredibly slow, since instead of access time measured in milliseconds over distances of hundreds of meters, expect access time requiring many seconds over several thousand kilometers. Figure out how to reduce/batch your DB calls/network IO, or mock them out (perhaps provide read-only copies on disk, and point your DB at them).

Machine setup/bootstrapping: What sort of configuration are you relying on, in terms of file setup/folder structure, and in terms of global state of the machine (environment variables, drive letters). Not building in any dependencies on global machine state ensure your tools are much more portable.

Localization/internationalization: Not every international outsourcer speaks English. Many places will translate documentation when settings things up, but that documentation can get stale. We should start thinking about writing localizable code and documentation.

Not so coincidentally- writing good code helps you make more portable tools. A lot of these problems aren’t really problems at all and they don’t really require extra work- they just require care and craftsmanship while developing your tools and code. Focusing on the critical path doesn’t give you a license to write bad code. In fact the contrary. The focus on the critical path means you should have excellent code at all times because the critical path is critical. So if you write good code, as you should, supporting outsourcers will be a lot easier when that time comes.

Do you have any experience or stories writing tools for outsourcers? I’d love to hear about them in the comments.

Refactoring is defined as a “disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior” by Martin Fowler.

But its everyday usage takes a very different meaning. We use the word refactor to everything from the original meaning, to a complete side-by-side rewrite of a barely-functional system. How can you use the same word to refer to something that does not change external behavior, with building an entirely new replacement system?

More than once, I’ve elicited sighs for my efforts to clarify the language we use. But when you have people that use wildly different vocabularies- artists, programmers, project managers- this is of paramount importance. So the fact that we say ‘refactor’ to meaning any type of rewriting of code or functionality irks me.

So recently I’ve begun a dictionary in my head for the type of tasks we do.

Refactor: The original and ‘precise’ meaning of restructuring a module without altering external behavior, up to a rewrite of parts of a larger system, that may change internal behavior of the system (including the external behavior of large internal components). While I’d love to keep only the more limited original meaning, when we’re dealing with large legacy codebases that often have zero tests, the original meaning doesn’t apply often enough.

Renovate: Rewriting a system so that its external behavior changes considerably but it still fulfills the same purpose (obviously). I name it as such because it is like renovating a building- the exterior and interior may greatly change, but what goes on inside may stay the same. Similar to ‘Rewrite’ but generally applies to a smaller (module/system) scale. An example would be, if you have a library for dealing with source control, and you no longer like the API. So you greatly change how the module works, and it still fulfills its fundamental purpose of dealing with source control.

Recycle: Writing a replacement system side-by-side with the old system, using components from the old system (by either referencing or copy/pasting), with the goal that once the new system is working, the old system will be shut down. The goal is a replacement that is easier to use but fulfills similar requirements. An example would be replacing legacy procedures for data transformations, that may have a lot of imperative code and poor reuse. You would replace the system with something better written, often taking chunks of logic from the old system, or writing tests that verify it produces the same results, then hook up calling code to use the new system, then delete the old one entirely.

Rewrite/Rebuild: When a system is to be written from the ground up, using the original implementation as an example or in a prototype role only, resulting in a system that fulfills the business requirements of the original system but does not necessarily preserve anything else about it. Similar to ‘Renovating’ but generally applies to a larger (application) scale. An example would be, if you have a website/game feature that you want to replace, you’d build the new one to fulfill the same business requirement (we need a website/some feature), but the features may be completely different.

So that’s my personal language right now. My hope is that it will expand to my team and then out from there. I don’t know if I’ll refine it much more- too granular and it would become unwieldy. Do you have suggestions or a language of your own?

My first big python project last year was yet another feed aggregator (taogreggator). Before I started, I looked around at what other aggregators were available, and wasn’t happy with any of them in terms of features, complexity, or trying to get each working.

Of course, 9 months later, that project is dead and I’ve successfully got the python ‘planet’ module up and running at www.tech-artists.org/planet.

Note, this blog post probably reveals what a big programming phony I am ;) Remember though that this sort of thing is well outside my usual domain of expertise.

So what happened? Why did it take so long to realize I was doing something stupid, regroup, and adopt something that actually works?

I was too ignorant to know better. Well, to be fair, I didn’t undertake this project out of hubris or to build something better, I built it mostly as a significant project I could train my python skills with.

I’m not interested in why it failed. There are 100 reasons why it failed, none of them unexpected or interesting. I’m interested in why I undertook it in the first place and took so long to trash it.

1. I didn’t know anything about the web

I still know barely anything, but trying to take an existing package and get it running was incredibly difficult, because I was so out of water. I didn’t even have the vocabulary, and was unfamiliar with everything I was supposed to do and the concepts of how things worked. My own project allowed me to get into it gradually.

2. Too inexperienced to know the challenges ahead of me

It wasn’t actually that difficult to get the app running locally. I even opened up a router port and ran my PC as a server, for remote connections. But I had an Ubuntu server to deploy to, and know nothing about Linux. I had never created a web app before. So at every step, I thought I was almost there. Every known was an unknown unknown to me, because I had no idea what to expect.

3. Too inexperienced with the commandline and the python environment

I talked about it in my Relearning Python series. When I started out, I didn’t really get how python works, because I came from .NET where I didn’t have to worry about any of that. I have a much, much better understanding now, and the environment is one of the early things I teach any new python programmer, because once you start importing code, or writing complex scripts, you need to know how it works. I didn’t understand the environment so I had a very difficult time getting any third-party systems set up.

4. Pythonic is more than a coding style

When I came to python, I was indoctrinated in the ways of a .NET programmer. It took me a long time to understand that ‘pythonic’ applies to more than just lines of code. It has to do with how you run your entire application. The way I run planet I’d consider entirely pythonic- I have a very thin script that generates and uploads some files. The planet module itself is pythonic- there’s some straightforward documentation, commented ini files, and templates, and you’re supposed to customize things and build a few wrapper scripts to run the stuff you need. This looseness was foreign, as I was more used to a much data-driven, rigid way of customizing an app. Being data driven is not great in all circumstances, especially when developing frameworks and apps like this, where the programmer is the user. When I saw what I ended up with with planet, I was embarrassed with how confusing my design was (though, to be fair, it had more features planned). Without understanding how I should use modules like planet, I couldn’t use them. Such basic stuff is not covered in a readme.

So, several weeks ago, I finally made an effort to deploy my custom aggregator on an AWS windows server. I still couldn’t get it working. And I was having even more questions about why I did stuff a certain way (I don’t think the code or design is particularly bad, but it made it difficult to use on a server). It was a huge failure. So three days later, after an awful day at work, I regrouped, and spent the entire evening figuring out existing aggregators, and after struggling with various ones, chose ‘planet’, and got pretty much everything working.

The lessons are pretty clear. You need some minimum knowledge to be able to make an informed decision. Attempt something of a very limited scope to give you that knowledge before making your decision. You will have plenty of options to reinvent the wheel when you know what you’re doing. On the other hand, if you’re pursuing a project only for educational purposes, do whatever you want :)

Next time I’m going to follow some tutorial end to end. It was fun hacking away on something way too complex, but I failed to deliver a server to the community, and, tbh, the time could have been better spent.

4 years, 5 branching strategies. First we worked all in one branch. Then we became hyper-branched. Then we consolidated into a couple branches. Switched companies. First we were all in one branch. Now we’re splitting into branches.

This has all been in Perforce since it is the de-facto SCM system for the games industry. But if we were using DCVS we’d probably have the same issues. The problem has not been merging changes. So DCVS is not the answer here (though I love DCVS).

I’ve been through this at two companies and have read about the experiences and strategies of other companies. I’ve found one constant across the differences in companies and strategies:

Branching strategy changes are in response to the instability that follows fast growth.

You cannot simply take a working model of how some project manages its branches, apply it to your studio, and be done with it. In fact, you cannot seek out or design an “ideal” branching strategy for your studio that is going to fix your instability problems. Why?

Branching is not designed to fix code instability.

Branching is a way to isolate changes and manage a release. It allows a much more flexible and intuitive use of version control by both developers and the studio, and allows sane release management. The DCVS branching model has proven itself and now we’re stuck trying to figure out how to get something similar in SCM systems like Perforce. But this is largely orthogonal to the problem of code instability.

You can keep unstable code in a branch, but it does nothing to fix the instability. You can require developers to run smoke tests, but they’re still going to integrate broken stuff, and they even get less ‘free QA’ while in their branch. We can put everyone on their own branch, or group teams on branches, or whatever strategy you want to come up with, and I don’t think any are guaranteed to work for your studio. Furthermore, studios change people and size, so what works one year may not work the next.

Yet we put so much effort into branching strategy as a way to solve these problems. We design a system for how the branches are laid out. We make some tools for creating and managing branches. We focus communication and training on how people people are supposed to work. Yet branching is not and should not be the way we actually fix the problems that caused the instability that caused us to change our strategy.

How do I know this? Because with every change in strategy, there is a much less prominent component at work.

Infrastructure and automated testing are coincidentally improved when we change branching strategies.

I don’t think anyone doesn’t consider these two things important for improving code stability. It is just that I think they’re almost totally responsible. I think that if you were to trace the successes of people’s branching experiments, they’d be completely dependent upon when their automated testing and infrastructure (like continuous integration and better messaging) turned a corner and became robust. So the fact that Strategy D worked is because the improvements to testing and infrastructure made from A to B, B to C, and C to D, have accumulated to where you have far less instability problems.

So what’s my beef with branching, or more specifically, changing strategies?

I don’t have any. I think there are, definitely, better and worse ways to do things. My problem is when we focus on branching strategies as the most important part of the instability solution. My problem is that we document, educate, build in order to support branching. We talk about “how we are going to be working in branches,” rather than “how we are going to build testable systems and get legacy code under test.”  We put our resources behind developing tools and fixing the fallout of branching, instead of making a focused education and cleanup effort towards getting things into a more testable state (which often includes the testing infrastructure as much as it means the application code).

Imagine if every time you heard ‘branching’ it was replaced with ‘testing/infrastructure,’ my guess is you’ve never heard managers talking about testing and infrastructure that much. Unfortunately you are unlikely to, because branching is an easy problem to think about. It is a chess board. No real work, personalities, real-world spikes. Just figuring out how to best move around your pieces in a theoretical way.

When you’re creating infrastructure, it isn’t a chess board. It is a world of incremental changes, no glamour, making do with the bare minimum, all on mission-critical systems that have countless tentacles. It isn’t the world of a plumber, it is the world of a septic tank diver.

But the real reason you’re not likely to see branching effort replaced with testing and infrastructure effort is because to do so can require a huge cultural and educational shift at a studio. Good luck teaching dozens of really smart developers who have decades of experience on successful projects that their code isn’t sufficient anymore, that you want to use your new fangled techniques that have actually proven successful in the rest of the development world. Those conversations aren’t why people become managers.

But mark my words, if you have a studio where testing is a fact of life, where it is not just an ideal but a requirement, where your infrastructure and developer systems are well understood, documented, extensible, and reliable, you are going to see very little code instability, regardless of what your branching strategy looks like.

If you’re thinking about changing how you branch, consider instead if all of that effort is spent on turning your codebase into something testable, your infrastructure and systems into something widely usable and reliable. If you want to achieve stability, you are going to have to do it anyway. The question is, do you do it as a side effect and keep taking the painful medicine of changing branches strategies to keep getting the side effect, or do you do the much more difficult thing in the short term and approach your instability problem head-on, through building, and creating a culture of, testing and infrastructure?