What I look for in a studio

by Rob Galanakis on 12/05/2011

The job hunt that landed me at CCP was an interesting experience.  It was long and difficult- but fun, because I knew exactly the type of job I wanted and type of studio I wanted to work at.  The job I wanted was a Lead Tech Art/Tech Art Director job, no surprises there.  Where I wanted to work, though, had much more going into it.

I’ve been very vocal that the large and nearly uncontrolled growth of BioWare Austin gave rise to most of the problems and incompatibilities I’ve developed with the studio.  Limitless people and money can have serious negative long term effects on any organization (even if it ensures the success of a single project).  So I look for a studio that can actively manage these issues as a healthy and stable place to work.

  1. How large does the studio plan to grow and how quickly?  If it plans to double in a year and potentially double again, I’m out.
  2. Where does the money to grow come from?  Is it self-financed, or does it come from a publisher?  If it comes from a publisher, I’m wary- third party finances can be arbitrary, and first party  money can create a scenario where:
  3. Is the project ‘too big to fail’?  Is the publisher going ‘all in’ on this project?  If so, too much money will be given to the project to reward incompetence (because the game needs to ship- see this year’s GDC lecture’s slides  about rewarding incompetence).
  4. Has the studio shipped a project of a similar size?  If the target project will need a target team size of 3x more than the next biggest project, be wary- no culture or studio can handle the complexities such growth for a single project brings.

All these factors lead towards a single point: Money breeds incompetence. Those are things that freak me out.  Here are the pluses:

  1. Has the studio shipped a successful game with principles I agree with?  Ie, I’m looking for a studio that has released an original, successful game, or has a track record of success on what it’s worked on.
  2. Is there a humility and stability to management?  Who am I meeting with for the interview, and where did the current management come from (are there ‘I’ve been at one company my whole life’ or are these ‘I shipped two failed MMO’s at different companies but it wasn’t my fault’ types, or something in the middle)?
  3. What’s the bonus plan like?  This is to gauge humility as much as the financial aspect.  Mostly, what’s the difference in percentage of salary between senior level developer and CEO?  OK: 0%/0%, 100%/anything, greater than/equal to a 1-to-5 ratio.  Not OK: Anything less than a 1-to-5 ratio.  I believe it shows a lack of care and respect for the people in the trenches when management gets such higher bonuses, in addition to the higher salary, more stock, executive perks, and the ability to manipulate bonuses (buying companies, getting rid of merit increases, etc.).

At CCP, I found a company that fit perfectly with what I was looking for.  It was the only company of the half a dozen I interviewed with that I felt completely comfortable taking a job with- I knew it was what I wanted, exactly what I was looking for.  It is great to feel like I’m going to a company that has ideals that align with mine, and I can have some confidence that those ideals won’t change in the near future (I hope I don’t eat those words).


From one space MMO to another

by Rob Galanakis on 10/05/2011

By now it is common knowledge amongst those that know me that I am leaving BioWare Austin and Star Wars: The Old Republic for a job at CCP Iceland as a Lead Technical Artist working on EVE and their Core Technology Group.

This job change also entails a change of roles, from a senior role to a leadership role, which I’m psyched about.  I’m taking a position where I can do the things regarding culture and vision that were difficult or impossible to do at BioWare because of my role and the culture there:  the studio is too caste-based for my tastes and I was outside of the management and leadership castes, which limited my ability to change anything.

I’m also stoked to be working for a studio like CCP- the single thing that attracted me most to the job, and convinced me that an international move would be worth it, is the studio culture.

In the coming months, I hope to talk more about both studio and tech art team culture issues, and the higher-level pipeline design issues, that are my passion and, hopefully, a large part of my new job.

Also, I’ll be in Austin late June, if you want to see me before I leave for Iceland.


Documentation is a tech possibility, not a project management hurdle

by Rob Galanakis on 8/05/2011

Months ago I made a post entitled “All About Documentation.”  In it, I discussed how in order to solve the documentation problem, we needed to either give up entirely, or make a serious attempt at solving it.  I mentioned I had some tech in the works, and Hamish McKenzie mentioned some details about his technical solution as well.  Well I’m here to say that I have finally implemented our ‘CollabDocs’ framework in our game tools, and holy cow is it awesome.

Here’s some info about it:

  1. Draws info from our Confluence wiki, but can be extended to draw from any source.
  2. Provides contextual information when user hovers over an enabled control.
  3. Tooltip can be clicked on to show the entire page, and that can be opened for edit or viewing in a browser.
  4. Info can be drawn from an entire wiki page, or just an area around an anchor.
  5. Can be set up entirely from Visual Studio’s Designer.
  6. Documents can be set to automatically reload on a timer, so data doesn’t get stale.
  7. We use this reloading mechanism to check if a page/section has changed, and can display alerts on our UI to alert the user if something has changed (some critical bug has been found, fixed, or new features have been rolled out).
  8. Any system can interface with the Confluence wiki and CollabDocs will pick it up- so build notes, for example, can be propagated to the proper place on the Wiki and will show up in the tools.

The truth is, I did the easy part- I was the initial advocate, set the vision and concept, and then implemented the framework into our tools (and added some very cool features, such as anchor support, better html rendering/parsing, the auto-reloading feature, bug fixes, etc.).  The hard work was done by Chris Long, a TA we hired in the Fall (around the time I made that initial post).  He is an absolutely awesome programmer and did a bang up job with his CollabDoc framework.

I’m really happy with how this all has come out, and it hasn’t been more than 2 days of work to get it integrated well into most of our tools.  And people are excited.  About documentation. Read those last two sentences again.  This may be the first time I’ve ever written those words (maybe the first time was when I discovered wiki software?  Well we know how that worked out).  I don’t know if CollabDoc is the end-all-be-all of documentation solutions.  In fact I know it’s not.  But it is a huge step forward.  And it shows that if you put your best and brightest on a problem (here’s looking at your, Chris), you can solve it in ingenious, sophisticated, and intelligent ways.  You just need those people, you need the vision to have them work on documentation, and then you need the drive to push it home (it took us months until I finally just did it myself).

I’ll have more details about how it works, and ideas on it, going forward as I hope to do something similar at my next job.  And I’ll also be submitting some sort of documentation-focused GDC session for GDC 2012.

No Comments

The Exception Bogeyman

by Rob Galanakis on 6/05/2011

I recently ran across a post by my colleague Chad Stewart, Lead Dev Support Engineer at BioWare Austin: Try/catch control flow

First of all, you ought to know that exceptions are deliberately named. That is to say, they are reserved for exceptional cases.

 this.SomeRiskyOperation(someParam, someLocal, -1);
catch (System.Data.SqlException)
 return true;
 return false;
return true;

This is sad because it is a big fallacy.  Exceptions as a concept have been around since well before modern programming languages, and they are for not at all exceptional cases in those modern languages (.NET and java are one implementation type, python would be another many are familiar with).  An ‘exception’ merely states that the invoked code could not execute normally.  It has nothing to do with being exceptional, or infrequent, or anything like that- it just indicates that code could not execute normally and cannot return a meaningful result.

If you think about something like a divide-by-zero exception- this is an issue of ambiguity.  What would you get if you divided a number by 0?  No one can say.  So what would you get if you tried to Frob and Sprocket but the Sprocket was null and there’s nothing to Frob?  Well, no one can say either- which is why null should be an invalid argument to Frob and Frob should throw an exception.  Likewise if the Frob method can only work on Sprockets that have been Blurmed.  If the Sprocket hasn’t been Blurmed, what’s Frob to do (unless its behavior in the case of a null or non-Blurmed Sprocket is clear)?

Similarly in the ‘not exceptional’ vein, python programmers have the ‘easier to ask for forgiveness than permission’ principle.  Rather than pre-validate everything, they try/except their way forward.  That is clearly a form of program flow, it is if/else by another name.

It’s sad that people don’t understand exceptions.  A large part of that is because exceptions are still thought of as exceptional.  Exceptions are not exceptional.  Not in modern managed languages they’re not, at least.  You need to understand how they work, how they are used, what they are for.  They are as important to program flow as any other language construct, they are just more complex.  There are best practices and ways to write robust code, there are also bad practices and it is easy to write bad code.  It is just more difficult to write good robust code when considering exceptions, because it is fundamentally more difficult (I’ll talk about exception handling and gotos in a future post, I promise).

Which is good, because most of the time, you don’t have to worry about exceptions (except if you use them like python programmers).  Which is good, because most of the time there’s nothing you can do about the exception.  Instead of worrying about exceptions, worry about writing good code, that is immutable, transactional, modular, testable, clear, explicit, etc.  I guarantee if you write better code, your exception handling experience will be better.  And exceptions reciprocate: code that enforces contract and behavior through exceptions will be more modular, testable, clear, and explicit.

So, back to Chad’s example code.  The problem with that code had nothing to do with exceptions. Rewrite that code without exceptions and it doesn’t get any better.  It may get faster, but the speed issue isn’t exceptions’ fault either- that is still just faulty code.  You can write slow code without exceptions (and I’ll bet your code spends infinitely more time in blocking IO than it does in exception handling).  The problem with that code is that it was shitty code. It was shitty because it wasn’t correct for its domain, which was inside of a tight loop.  There’s honestly nothing so conceptually wrong with that code from the sample- if SomeRiskyOperation is free of side effects, and the containing method has a clear true/false contract.  But I don’t have enough context to judge.  I am just saying that that code  which looks horrible may very well be fine.  It is just an example of the exception bogeyman, that uncomfortable programmers use to terrorize younger programmers that will listen, and it is a huge lie.

This isn’t the last you’ll hear from me about exceptions.


Game designers vs. Tools designers

by Rob Galanakis on 4/05/2011

(I wrote this post in November 2010 and it was an important influence for my GDC2011 talk.  I’m finally posting it.  See the GDC slides/notes for much deeper discussion).

There’s a problem at many studios, I think, where our lead game designers are also responsible for designing tools and design workflows. A tools programming team is then responsible for implementing it. Where I’ve seen this done, I think it has been a catastrophic disaster, and I’m sure it is this way at many studios. This is a bad model for so many reasons. The right model is the ‘tech art’ model, which I’ll examine the pros and cons of more fully later.

Only designers know what they have to do, only engineers can explore what is possible
An engineer can only design or implement a mediocre tool if he doesn’t have a deep understanding of a workflow. A designer cannot see the big picture for issues outside of his workflow. I’ve seen tools developers be able to eliminate entire workflows by understanding not just how, but why something needs to be done, understanding how it ties into another area, and completely eliminating a workflow because it becomes redundant when harnessing that other area. Solving this requires a highly iterative development process, designers writing tools, or engineers embedded with the design team.

Game design is conceptual, tools design is technical
I am not game designer, so apologies if I’m over-simplifying. But I see game design as creating and implementing a vision. And while the best designers are both very conceptual and excellent technicians (as the best tools developers are), the technical skills are in support of a known end concept (or in support of figuring out what the end concept is). Tools design, on the other hand, is about identifying and fixing problems- it is not about creating and implementing a vision. While there can certainly be tools visions (like the object model pipeline!), that is at a comparable level to choosing a game’s genre- it is merely a framework for decisions and not a normal decision in and of itself.

Ignoring people’s strengths
Simply, if we accept that the relation between tools and game design is tenuous at best, you are ignoring people’s strengths by using the same person to head up both. Game designers should be masters of vision and tools designers need to be able to see how to execute the vision.

Competition for resources breeds adversarial relationships
If the people designing and using the tools are not able to manage their own resources and execute on their own needs, an adversarial relationship will develop between design, engineering, and people competing for engineering resources. Engineers will not be as invested in their work because they have no investment in it, and their closest mates have no investment in it. Design will complain about mediocre tools and trust engineering less.

Which breeds short-term thinking
If neither side truly understands the other, and neither side really wants to work with the other, phrases like ‘we won’t need to change that’ are uttered and believed much more often. Design is idiotic for thinking they won’t need to change something; engineering is idiotic for believing it. Design doesn’t ‘see the big picture’ and realize they will need to change it; engineering will ‘give them what they ask for’ because that is all they are supposed to do- they have no investment.

So what do we do about it?  We can adopt the Tech Art Team Model for the development of design tools.  More details in future posts.

No Comments

Relationship between PMs and Dev happiness

by Rob Galanakis on 1/05/2011

Consider this thought:

As the ratio of Project Managers to Developers increases, the happiness of Project Managers generally rises and the happiness of the Developers generally decreases.

1 Comment

Two keys to effective multithreaded programs

by Rob Galanakis on 29/04/2011

In my last post, I went over a mostly lock-free producer/consumer queue that worked entirely off the ThreadPool. This covers the two most important aspects to effective multithreaded programming: avoid creating new Threads, and work lock-free where possible.

Avoiding the creation of new threads is easy- you just need to schedule tasks on the ThreadPool (though QueueUserWorkItem, or System.Threading.Tasks usage), instead of using Thread.Start or new Thread. The goal is, you want the ThreadPool to manage threads for optimal performance, you don’t want to create other Threads that are going to push the ThreadPool threads off the CPU, and are going to need to be created/destroyed. A long-lived thread is usually a waste because it is inactive for much of the time. A short-lived thread is a waste because allocating and destroying a thread is an expensive operation! So just avoid creating threads- you almost never have to if you design your systems properly.

Lock-free programming is much more difficult. The reason lock-free programming is important (in this case) is, you don’t want to block ThreadPool threads. If a ThreadPool thread is blocked for a while, the ThreadPool will create a new thread- which means we’re basically creating new Threads as above, which is bad! Worst case scenario: a Parallel.ForEach loop that involves the launching of a Process/WaitForExit that blocks the thread. You’ll end up with almost every iteration of the ForEach loop having launched its own process, so you’ll have n threads in your main program, with n processes running, with their own threads. You will have a huge mess and everything will be context switching like crazy and performance across the board will suffer!

So this all basically requires breaking sequential work into discrete chunks, and linking them together- and actually this is exactly what Task Based Parallelism is, and you can do it effectively with System.Threading.Tasks in .NET 4.0 (or 3.5 with Reactive Extensions’ System.Threading.dll), and with .NET 5.0′s async and await keywords.  However, these are necessarily high-level concepts and systems, and you cannot just throw code into these systems and expect them to work correctly or predictably.  Managing threads effectively, as described here, is the first (and easiest) component of writing effective asynchronous/multithreaded systems.  The more difficult part is writing systems that can function correctly and predictably in a asynchronous environment.  I’ve found, though, that thinking about the (easier) thread management aspect can help inform the (difficult) systems design component.


Completely async file writer

by Rob Galanakis on 26/04/2011

I’ve been doing more and more asynchronous programming lately.  I needed to implement a logging system recently, but wanted to use asynchronous file IO.  There was also the possibility that log calls could come in faster than they could complete, so I needed to synchronize the logging calls, marking them finished when the async IO completed.  I wanted to make this as lock-free as possible.

The idea is, you have a ‘AsyncFileWriter’ field on a class, with a ‘QueueWrite’ method that threads can call to request a file write, passing the string/bytes, and a callback that fires when the IO is complete.

The ‘AsyncFileWriter’ has a Queue on it, and when calls to QueueWrite are made, a request (args and callback) is added to the Queue, and a call to a ‘WriteImpl’ is queued on the ThreadPool.  ’WriteImpl’ has basically a semaphore, only allowing 1 thread to dequeue a request (args and callback) and begin the file IO.  This ‘semaphore’ is released in the async IO complete callback.  After the semaphore is released, another call to WriteImpl is queued on the ThreadPool, which will just return immediately if the request queue is empty.

The net effect is we have, basically, a producer-consumer queue, where producer threads can also be consumers (or queue up other ThreadPool threads to be consumers).  When a request is processed, the consumer thread that finished processing it queues up another consumer thread.  If a request comes in while the ‘semaphore’ is taken, the thread just drops its request into the request Queue and immediately returns- if the ‘semaphore’ is not taken, the thread dropping off the request then takes the lock and acts as a consumer thread.

I think the pseudocode is more clear than the actual code:

class AsyncFileWriter
   string filename #initialized from constructor
   Queue _requestQueue
   bool _insideWrite

   public void QueueWrite(byte[] bytes, Action callback)
      lock (_requestQueue) _requestQueue.Add(bytes, callback)

   private void WriteImpl()
      if (_insideWrite)
      lock (_requestQueue)
         request = _requestQueue.Dequeue()
         _insideWrite = true
      if request != null:
         fs = new FileStream(filename)
         fs.BeginWrite(request.Bytes, () => OnIOComplete(fs, request.Callback))

   private void InIOComplete(FileStream fs, Action callback)
      _insideWrite = false
      ThreadPool.QueueWork(WriteImpl) #Keep clearing the queue

So you can see that, we only take locks when adding or removing items from the queue. If 5 threads enter WriteImpl at the same time, only one will take the lock, and the rest will return. And when the IO is complete, it’ll queue up another ThreadPool request to keep clearing the queue.

I initially implemented this for async IO as mentioned above, but have abstracted the pattern for anything- it is just basically a way to throttle processing with a ThreadPool-managed producer/consumer queue. I’ll go over this pattern along with some code in a future post.

1 Comment

Do you have your Visual Studio WinForms Designer License?

by Rob Galanakis on 23/04/2011

I often joke at work that you should need a license to use the Visual Studio GUI Designer.  Not because it is hard to use- quite the contrary.  Because it is so goddamn easy to make a UI, people forget that UI classes should be written like any other piece of code, NOT as if they were created using a WYSIWYG editor.

Since when was it OK to have an object with 70 fields?  Why is OK to have a UserControl with 70 fields on it, including a context menu and top menu, a tab control and three full tab pages set up on it?

I’ve seen very talented programmers forget that they are talented programmers, and build UI’s with the designer that are absolutely atrocious.  Here’s how I’d suggest you break bad habits for using the Designer:

  1. Only use the Designer to set up UserControl subclasses of no more than 10 controls (both Controls and other subclassed UserControls).  If your UserControl/Form subclass has more than 10 designer-fields, it’s too big.
  2. All ToolStripItem/MenuItem classes should always be created dynamically (or at least in the main class file), NOT through the Designer.  Remember, you can build a ‘tree’ of ToolStripItems through code (think of Linq2Xml and how the code formally looks like the XML), not having to assign things to fields all over the place.  I’ll show an example in a future post.
  3. All events should be assigned through code, NOT the Designer (you can also avoid tons of event handler methods by just hooking up the events with lambdas that call the method you would have been calling).
  4. Use subclasses of Control types (ComboBox, ContextMenuStrip) to encapsulate logic.  Make them private classes inside of your UserControl if possible- the goal is to break up the logic out of a ton of methods on the UserControl.  If you have more than 4 control-specific methods, look at creating a subclass.
  5. Rarely if ever pass a reference to a Parent around- instead, raise events on your children and listen to them from the parent.
  6. If you can create your UI dynamically, you can get away with private parameterless constructors on UserControls, so you can actually have some data integrity and straightforward object use (no ‘must call Initialize/Bind method to assign data objects’ crap).
  7. Learn UI programming in a language without a WYSIWYG editor, like python.  You quickly learn small UI classes laid out in a dynamic manner are more flexible than monster static UI’s that the VS Designer is often used to create.

I can quickly gauge how passionate and experienced someone is about their craft by looking at how they build a complex UI.  If they build designer-driven uber-classes, you can be sure they either haven’t had enough experience building complex UI’s, or haven’t thought enough about what they’re doing to realize how fucked up the code they are writing is.

1 Comment

Get better at coding or get better at arguing

by Rob Galanakis on 21/04/2011

A while ago, one of the junior TA’s here was frustrated with all the code critiquing I was giving him, and exclaimed: “In 6 months, we haven’t had an argument about my code which I’ve won.”

I replied, “Well, get better at coding or get better at arguing.”

He got better at both.

No Comments

Switch to our mobile site