The right kind of consistency

by Rob Galanakis on 22/05/2011

Code consistency seems like a straightforward topic.  And it is, when you consider just the code.  But you have to consider the team behind the code and the long term evolution of the codebase.  The best architects will enforce consistency in the right area at the right time.

The ‘time’ aspect is particularly important.  Consistency at the current instant in time is important.  The team as a whole should be of a consistent mind with those things that are agreed to be consistent.  But what is ultimately a huge negative is consistency over a long period of time.

This problem manifests itself from minor to major consistency issues.  On the minor end, let’s say you prefix all members with ‘m_’.  You realize later than ‘_’ is shorter and much easier to type.  Why not just switch over to ‘_’, if your team agrees it is better (and who wouldn’t?).  If everyone agrees, then you don’t have to worry about ‘competing styles’, as often happens if you don’t have a uniform style.  You’ll have more code edits as you change over styles, sure- but these are minor and trivial edits (and very easy to diff/resolve/merge, if that’s what you’re worried about).

On the more major end, you have something like not allowing early returns.  Again, there’s no reason to enforce this rule for the sake of consistency- debugging with early/multiple returns is no harder.  What is clear, though, is coding with less nesting is a benefit (nesting increases the cognitive burden placed on the reader since a single line of code seems actively dependent on more states- there’s good writing about this if you want to investigate).  But mostly, having one return in a method is another outdated C++ leftover.  So why sacrifice all new code because the old code, conceived in a more ignorant time, is inefficient?  Break consistency to move forward.

And that’s the key to the whole thing.  Consistency can stand in the way of progress. If current conventions are tenuous, don’t be opposed to changing them for the sake of consistency.  It is the consistency of the team and not the consistency of the codebase that matters. Otherwise, you end up only evaluating and improving conventions every several years, instead of every several months.  More rapid convention changes means you have more changes and chances to see what works and what doesn’t, find new things out, and move your codebase and team forward, inch by inch.

No Comments

A lead shouldn’t lead like lead.

by Rob Galanakis on 19/05/2011

I’ve been thinking a lot about what I’m going to do with my new role at CCP.  Most of these I’ve been able to do (or advocate strongly for) to some degree even as a non-Lead but they were never supported enough by the Leads and never took hold.

  1. Training sessions.  These are vital for the health of the team in so many ways.  It exposes people to new ideas, stimulates discussion, and gets people thinking about things differently.  It also gets them contributing to the social aspect of the ‘team’ rather than just contributing files into a codebase.
  2. Collective vision and self-tasking.  I was always most frustrated when I didn’t have a say in what I was working on.  True scrumming alleviates this, but even without scrum, team members need to decide what the team is going to be working on as a whole and what they are going to take on in particular.
  3. Changing concepts/standards/practices.  Consistency in a codebase is great, but keeping a team motivated and passionate is more important, and will result in better code overall.  I think the current concepts/standards/practices should be consistent, but they need to change over time.  Developers change, and new ones join, and they have different opinions and better ideas.  You shouldn’t frustrate them by stifling superior ideas for the sake of consistency or resistance to change.
  4. Community engagement.  I don’t think you can be an excellent developer in this day and age unless you are part of the coding community- force everyone to have a Stack Overflow account and come up with a blog bundle for the team.  And then have a ‘reading group’ every week or two to cover interesting concepts.
  5. Move people around.  Never have your entire team sitting together permanently, and never embed a team member with a team permanently.  4-6 months at a time, and switch things up.  Expose your tool devs and TA’s to different groups, to get more context and develop more relationships, and then move them ‘back home’ to refine and refresh their skills.
  6. Code Katas.  Another team-building exercise that can easily be factored into training sessions.  Also useful to cross-pollinate teams, so programmers from different teams can pair and compete against their buddies.

The key here is: change.  As a lead, you are your teams biggest enemy.  You have the power to grow your teammates and the team, and you have the power to frustrate teammates and stunt growth, more than anyone on the team (you have the most power on the team, and with great power…).  You need to make sure the team doesn’t stagnate, and the way you need to do that is by integrating mechanisms to guide change into your every day interactions.

And in a year, I’ll be able to look back at this post and evaluate how well I executed on what I wanted to do- and if something is ill advised I’ll be able to judge why I didn’t realize it.


Cars, by definition, cannot fly

by Rob Galanakis on 17/05/2011

Saying that you need to throw away years of work and start from scratch is rarely a practical solution.  And I absolutely agree, and am a huge supporter of trying to make solutions that work in context and are practical.  But there must be times when incremental improvements must be abandoned for the sake of a new direction.

You cannot make a car fly by putting a bigger engine in it or making it lighter.  The fundamentals of its design are different from a device designed to fly.  When you are figuring out whether you should salvage existing decisions and incrementally improve, or if you should throw it away and start over, you need to ask yourself- are you trying to turn a car into a plane?

Sometimes we have cars and all we need are faster cars.  And sometimes we have a plane and we just need a safer plane.  But sometimes we have a car and imagine if we keep improving the engine and frame, we can make it fly.  We can’t.

But it is also important that when you decide you want a plane instead of a car, you don’t just abandon all ground vehicles.  We’ve been flying for almost a century, but we all still drive cars.  When making these huge technological leaps forward, we need to remember how risky the approach is and make sure what we do is as backwards compatible as possible.

This is an always risky and passionate issue but lately I’ve been using this as a useful analogy.  Identifying the fundamental problems is always the biggest obstacle to implementing a vision, and first and most difficult is to identify and differentiate the truly fundamental problems from the merely difficult to solve.

No Comments

Async IO- don’t do naive async!

by Rob Galanakis on 14/05/2011

This post comes from a response here: .  The second-highest rated response includes absolutely terrible advice.  It says, just put the file IO on a background thread.  It shows a complete lack of understanding of how IO works.

When your do IO (reading/writing from/to HDD or network), the software thread goes to the hardware, and says, ‘hey, can you write this info somewhere’ or ‘hey, I need this info, can you get it for me.’  The hardware then takes that request and does something with it, and when it is done it tells the software thread (let’s assume there’s no caching or lazy behavior going on).  This entire time, the software thread is just waiting while the hardware does the work.

What you really want to do is have the software thread run to the hardware and say, ‘hey, can you do this for me, I’ll be back later,’ then run back to the software threadpool (where all the inactive software threads hang out).  And then it should be dispatched to do something else (like run another request down to the hardware, or some processing).  When the hardware is done, the software threads will pick it up and report it as finished- the software threads are never waiting around for hardware to do stuff.

Unfortunately async IO in .NET is cumbersome, so it is often not worth doing truly async IO.  So it still makes sense to do IO on background threads, but it is just inefficient and NOT best practice- hence this post.  Just keep in mind this is a vastly simplified version of what actually goes on, but it is important to understand it at least on some basic level so you can design efficient and fast systems.

No Comments

What I look for in a studio

by Rob Galanakis on 12/05/2011

The job hunt that landed me at CCP was an interesting experience.  It was long and difficult- but fun, because I knew exactly the type of job I wanted and type of studio I wanted to work at.  The job I wanted was a Lead Tech Art/Tech Art Director job, no surprises there.  Where I wanted to work, though, had much more going into it.

I’ve been very vocal that the large and nearly uncontrolled growth of BioWare Austin gave rise to most of the problems and incompatibilities I’ve developed with the studio.  Limitless people and money can have serious negative long term effects on any organization (even if it ensures the success of a single project).  So I look for a studio that can actively manage these issues as a healthy and stable place to work.

  1. How large does the studio plan to grow and how quickly?  If it plans to double in a year and potentially double again, I’m out.
  2. Where does the money to grow come from?  Is it self-financed, or does it come from a publisher?  If it comes from a publisher, I’m wary- third party finances can be arbitrary, and first party  money can create a scenario where:
  3. Is the project ‘too big to fail’?  Is the publisher going ‘all in’ on this project?  If so, too much money will be given to the project to reward incompetence (because the game needs to ship- see this year’s GDC lecture’s slides  about rewarding incompetence).
  4. Has the studio shipped a project of a similar size?  If the target project will need a target team size of 3x more than the next biggest project, be wary- no culture or studio can handle the complexities such growth for a single project brings.

All these factors lead towards a single point: Money breeds incompetence. Those are things that freak me out.  Here are the pluses:

  1. Has the studio shipped a successful game with principles I agree with?  Ie, I’m looking for a studio that has released an original, successful game, or has a track record of success on what it’s worked on.
  2. Is there a humility and stability to management?  Who am I meeting with for the interview, and where did the current management come from (are there ‘I’ve been at one company my whole life’ or are these ‘I shipped two failed MMO’s at different companies but it wasn’t my fault’ types, or something in the middle)?
  3. What’s the bonus plan like?  This is to gauge humility as much as the financial aspect.  Mostly, what’s the difference in percentage of salary between senior level developer and CEO?  OK: 0%/0%, 100%/anything, greater than/equal to a 1-to-5 ratio.  Not OK: Anything less than a 1-to-5 ratio.  I believe it shows a lack of care and respect for the people in the trenches when management gets such higher bonuses, in addition to the higher salary, more stock, executive perks, and the ability to manipulate bonuses (buying companies, getting rid of merit increases, etc.).

At CCP, I found a company that fit perfectly with what I was looking for.  It was the only company of the half a dozen I interviewed with that I felt completely comfortable taking a job with- I knew it was what I wanted, exactly what I was looking for.  It is great to feel like I’m going to a company that has ideals that align with mine, and I can have some confidence that those ideals won’t change in the near future (I hope I don’t eat those words).


From one space MMO to another

by Rob Galanakis on 10/05/2011

By now it is common knowledge amongst those that know me that I am leaving BioWare Austin and Star Wars: The Old Republic for a job at CCP Iceland as a Lead Technical Artist working on EVE and their Core Technology Group.

This job change also entails a change of roles, from a senior role to a leadership role, which I’m psyched about.  I’m taking a position where I can do the things regarding culture and vision that were difficult or impossible to do at BioWare because of my role and the culture there:  the studio is too caste-based for my tastes and I was outside of the management and leadership castes, which limited my ability to change anything.

I’m also stoked to be working for a studio like CCP- the single thing that attracted me most to the job, and convinced me that an international move would be worth it, is the studio culture.

In the coming months, I hope to talk more about both studio and tech art team culture issues, and the higher-level pipeline design issues, that are my passion and, hopefully, a large part of my new job.

Also, I’ll be in Austin late June, if you want to see me before I leave for Iceland.


Documentation is a tech possibility, not a project management hurdle

by Rob Galanakis on 8/05/2011

Months ago I made a post entitled “All About Documentation.”  In it, I discussed how in order to solve the documentation problem, we needed to either give up entirely, or make a serious attempt at solving it.  I mentioned I had some tech in the works, and Hamish McKenzie mentioned some details about his technical solution as well.  Well I’m here to say that I have finally implemented our ‘CollabDocs’ framework in our game tools, and holy cow is it awesome.

Here’s some info about it:

  1. Draws info from our Confluence wiki, but can be extended to draw from any source.
  2. Provides contextual information when user hovers over an enabled control.
  3. Tooltip can be clicked on to show the entire page, and that can be opened for edit or viewing in a browser.
  4. Info can be drawn from an entire wiki page, or just an area around an anchor.
  5. Can be set up entirely from Visual Studio’s Designer.
  6. Documents can be set to automatically reload on a timer, so data doesn’t get stale.
  7. We use this reloading mechanism to check if a page/section has changed, and can display alerts on our UI to alert the user if something has changed (some critical bug has been found, fixed, or new features have been rolled out).
  8. Any system can interface with the Confluence wiki and CollabDocs will pick it up- so build notes, for example, can be propagated to the proper place on the Wiki and will show up in the tools.

The truth is, I did the easy part- I was the initial advocate, set the vision and concept, and then implemented the framework into our tools (and added some very cool features, such as anchor support, better html rendering/parsing, the auto-reloading feature, bug fixes, etc.).  The hard work was done by Chris Long, a TA we hired in the Fall (around the time I made that initial post).  He is an absolutely awesome programmer and did a bang up job with his CollabDoc framework.

I’m really happy with how this all has come out, and it hasn’t been more than 2 days of work to get it integrated well into most of our tools.  And people are excited.  About documentation. Read those last two sentences again.  This may be the first time I’ve ever written those words (maybe the first time was when I discovered wiki software?  Well we know how that worked out).  I don’t know if CollabDoc is the end-all-be-all of documentation solutions.  In fact I know it’s not.  But it is a huge step forward.  And it shows that if you put your best and brightest on a problem (here’s looking at your, Chris), you can solve it in ingenious, sophisticated, and intelligent ways.  You just need those people, you need the vision to have them work on documentation, and then you need the drive to push it home (it took us months until I finally just did it myself).

I’ll have more details about how it works, and ideas on it, going forward as I hope to do something similar at my next job.  And I’ll also be submitting some sort of documentation-focused GDC session for GDC 2012.

No Comments

The Exception Bogeyman

by Rob Galanakis on 6/05/2011

I recently ran across a post by my colleague Chad Stewart, Lead Dev Support Engineer at BioWare Austin: Try/catch control flow

First of all, you ought to know that exceptions are deliberately named. That is to say, they are reserved for exceptional cases.

 this.SomeRiskyOperation(someParam, someLocal, -1);
catch (System.Data.SqlException)
 return true;
 return false;
return true;

This is sad because it is a big fallacy.  Exceptions as a concept have been around since well before modern programming languages, and they are for not at all exceptional cases in those modern languages (.NET and java are one implementation type, python would be another many are familiar with).  An ‘exception’ merely states that the invoked code could not execute normally.  It has nothing to do with being exceptional, or infrequent, or anything like that- it just indicates that code could not execute normally and cannot return a meaningful result.

If you think about something like a divide-by-zero exception- this is an issue of ambiguity.  What would you get if you divided a number by 0?  No one can say.  So what would you get if you tried to Frob and Sprocket but the Sprocket was null and there’s nothing to Frob?  Well, no one can say either- which is why null should be an invalid argument to Frob and Frob should throw an exception.  Likewise if the Frob method can only work on Sprockets that have been Blurmed.  If the Sprocket hasn’t been Blurmed, what’s Frob to do (unless its behavior in the case of a null or non-Blurmed Sprocket is clear)?

Similarly in the ‘not exceptional’ vein, python programmers have the ‘easier to ask for forgiveness than permission’ principle.  Rather than pre-validate everything, they try/except their way forward.  That is clearly a form of program flow, it is if/else by another name.

It’s sad that people don’t understand exceptions.  A large part of that is because exceptions are still thought of as exceptional.  Exceptions are not exceptional.  Not in modern managed languages they’re not, at least.  You need to understand how they work, how they are used, what they are for.  They are as important to program flow as any other language construct, they are just more complex.  There are best practices and ways to write robust code, there are also bad practices and it is easy to write bad code.  It is just more difficult to write good robust code when considering exceptions, because it is fundamentally more difficult (I’ll talk about exception handling and gotos in a future post, I promise).

Which is good, because most of the time, you don’t have to worry about exceptions (except if you use them like python programmers).  Which is good, because most of the time there’s nothing you can do about the exception.  Instead of worrying about exceptions, worry about writing good code, that is immutable, transactional, modular, testable, clear, explicit, etc.  I guarantee if you write better code, your exception handling experience will be better.  And exceptions reciprocate: code that enforces contract and behavior through exceptions will be more modular, testable, clear, and explicit.

So, back to Chad’s example code.  The problem with that code had nothing to do with exceptions. Rewrite that code without exceptions and it doesn’t get any better.  It may get faster, but the speed issue isn’t exceptions’ fault either- that is still just faulty code.  You can write slow code without exceptions (and I’ll bet your code spends infinitely more time in blocking IO than it does in exception handling).  The problem with that code is that it was shitty code. It was shitty because it wasn’t correct for its domain, which was inside of a tight loop.  There’s honestly nothing so conceptually wrong with that code from the sample- if SomeRiskyOperation is free of side effects, and the containing method has a clear true/false contract.  But I don’t have enough context to judge.  I am just saying that that code  which looks horrible may very well be fine.  It is just an example of the exception bogeyman, that uncomfortable programmers use to terrorize younger programmers that will listen, and it is a huge lie.

This isn’t the last you’ll hear from me about exceptions.


Game designers vs. Tools designers

by Rob Galanakis on 4/05/2011

(I wrote this post in November 2010 and it was an important influence for my GDC2011 talk.  I’m finally posting it.  See the GDC slides/notes for much deeper discussion).

There’s a problem at many studios, I think, where our lead game designers are also responsible for designing tools and design workflows. A tools programming team is then responsible for implementing it. Where I’ve seen this done, I think it has been a catastrophic disaster, and I’m sure it is this way at many studios. This is a bad model for so many reasons. The right model is the ‘tech art’ model, which I’ll examine the pros and cons of more fully later.

Only designers know what they have to do, only engineers can explore what is possible
An engineer can only design or implement a mediocre tool if he doesn’t have a deep understanding of a workflow. A designer cannot see the big picture for issues outside of his workflow. I’ve seen tools developers be able to eliminate entire workflows by understanding not just how, but why something needs to be done, understanding how it ties into another area, and completely eliminating a workflow because it becomes redundant when harnessing that other area. Solving this requires a highly iterative development process, designers writing tools, or engineers embedded with the design team.

Game design is conceptual, tools design is technical
I am not game designer, so apologies if I’m over-simplifying. But I see game design as creating and implementing a vision. And while the best designers are both very conceptual and excellent technicians (as the best tools developers are), the technical skills are in support of a known end concept (or in support of figuring out what the end concept is). Tools design, on the other hand, is about identifying and fixing problems- it is not about creating and implementing a vision. While there can certainly be tools visions (like the object model pipeline!), that is at a comparable level to choosing a game’s genre- it is merely a framework for decisions and not a normal decision in and of itself.

Ignoring people’s strengths
Simply, if we accept that the relation between tools and game design is tenuous at best, you are ignoring people’s strengths by using the same person to head up both. Game designers should be masters of vision and tools designers need to be able to see how to execute the vision.

Competition for resources breeds adversarial relationships
If the people designing and using the tools are not able to manage their own resources and execute on their own needs, an adversarial relationship will develop between design, engineering, and people competing for engineering resources. Engineers will not be as invested in their work because they have no investment in it, and their closest mates have no investment in it. Design will complain about mediocre tools and trust engineering less.

Which breeds short-term thinking
If neither side truly understands the other, and neither side really wants to work with the other, phrases like ‘we won’t need to change that’ are uttered and believed much more often. Design is idiotic for thinking they won’t need to change something; engineering is idiotic for believing it. Design doesn’t ‘see the big picture’ and realize they will need to change it; engineering will ‘give them what they ask for’ because that is all they are supposed to do- they have no investment.

So what do we do about it?  We can adopt the Tech Art Team Model for the development of design tools.  More details in future posts.

No Comments

Relationship between PMs and Dev happiness

by Rob Galanakis on 1/05/2011

Consider this thought:

As the ratio of Project Managers to Developers increases, the happiness of Project Managers generally rises and the happiness of the Developers generally decreases.

1 Comment