Blog of Rob Galanakis (@robgalanakis)

Posts by: Rob Galanakis


Jim Crow is hiding in Silicon Valley’s hiring

What do Silicon Valley interview processes have in common with Jim Crow literacy tests? A surprising amount!

Read more


And now, some terrible advice

In a previous post, I talked about the most important advice I’d ever gotten. Now, lest you think some people are really wise and just dispense good advice constantly, I’ll tell you about how the same individual also gave some terrible advice.† The advice can be distilled down to this: If you are a manager and still doing individual work, you’re not doing your job as a manager. This was established at the top, and percolated all the way down to managers who had one direct report. The advice is not universally wrong, but it leaves me with questions. I honestly don’t know if Gordon intended to mean that a Team Lead with one report was not expected to be able to execute individual contributor (IC) work at a high level. But that was the effect and BioWare was full of such leads. Ambitious people mimic their superiors. This seems to be true even for the independently-minded ones. If we flaunt inferior IC skills as a status symbol of our rank, we should expect those ambitious people to actually revere technical incompetence. Lack of IC skill becomes important evidence that someone is ready for a promotion. There’s an obvious conundrum here: you do end up doing less IC work as you take on more management responsibilities. Your IC skills will degrade. And managers not letting go of IC work is a problem. But that is exactly why Gordon’s advice here is so terrible. It focuses on theater, and provides no grounding principle or value. Rather than helping people cope with the very real difficulties of moving into various levels of management, this mantra I heard so many times told them to ‘act the part’ of what he considered a successful manager. In my opinion and experience, successful management will...

Read more


How the Zapier interview process is making me rethink my own

Over the last two months of looking for a new job, I’ve applied to over a couple dozen companies and interviewed with about a quarter of them. The process that stood out the most was Zapier’s. Their interview process is non-typical from start to finish. They have a great blog post describing it, but here’s a quick overview: Applying: Multiple essay and formal questions. Evaluation: Zapier’s hiring team evaluates each application, which can take a few weeks. Job Fit Interview: One hour interview with the hiring manager. Skills Interview: Each position has one or more skills evaluations done by the hiring team, such as a programming or leadership exercise. Final Decision/Reference Checks/Offer are what you’d expect. Rejection (if applicable): Zapier sends you a rejection email. I was rejected after the Job Fit interview, so unfortunately I’m not able to comment on later steps. But my three interactions with Zapier (Applying, Job Fit, Rejection) were all very well thought out and I’d like to go through the ingenuity and benefit of each one. Applying Normally, companies ask for a cover letter and a resumé. This is unfortunate, as these are very low-signal, arbitrary, and prone to excessive bias: There is no universal agreement on what a resumé or cover letter should contain or how they should be written. The quality of a cover letter and resumé reflects the degree of coaching, rather than quality of the individual. Most hiring managers will have a particular style of resumé and cover letter they prefer. This preference is based primarily on pattern matching, and is opaque to the application process, which both contribute to systemic bias. Zapier gets around these problems by asking over a half dozen long-form questions (my application was almost six pages, each question being basically a short blog post)....

Read more


Failed assertions and async functions

Armin Ronacher asked in a tweet: If you want to signal a bad calll from an async function (failed assertion). Do you … — Armin Ronacher (@mitsuhiko) February 7, 2018 I explored this quite a bit working in JavaScript on the client and feel like I have a good understanding of the tradeoffs (tl;dr is to return a rejected promise with an Error, but I explain how we made this work well). First off, I’m usually very strict about erroring/raising/throwing when a programmer error happens, like passing in an invalid value. I want programmer errors to raise loudly, as early as possible, so they can be fixed. This is originally the approach I went with at Cozy. However, most of the event handlers that called async code had a form like: turnOnLoadingState(); doAsyncThing(args). tap(showSuccess). tapCatch(showError). // See http://bluebirdjs.com/docs/api/tapcatch.html finally(turnOffLoadingState); If doAsyncThing raised an error (asserted), turnOffLoadingState would never be called, and the loading state would continue to be active. The perpetual loading state is to me the most unacceptable user state. I would much rather have a user rage-clicking a button than I would show them an inactive screen or button that is ‘still processing.’ An obviously broken button is obvious; the user can bug out or reload or whatever once they give up. A stuck loading state begs the user to hang on until they get frustrated and leave, unsure if what they did ever worked, and what the state of the world will be when they try again. Failed assertions were triggered too often, usually due to the chaos, asynchrony, and unpredictable behavior of browser JavaScript and the DOM. The story may be different using certain frameworks or under certain conditions, but in real world browser code, I suspect not (hello browser extensions!). To avoid a perpetual...

Read more


Diffusing religious arguments

I called arguments between assemble-your-own vs. kitchen-sink framework approaches a religious one akin to tabs vs. spaces in my previous post. All of my leadership positions have been on brownfield products, usually turnarounds, so I’ve walked into minefields of religion that I’ve had to diffuse in order to focus on fixing the serious issues. Some of the topics include multiple conflicting code styles in a single codebase; internal code styles that conflict with the official (or most popular) language coding styles; choice of a third-party library; strict vs. lenient linting and what to lint; and competing internal libraries or services that do the same thing. I’m not sure what type religious argument I haven’t run into, now that I think about it. The process for resolving any of these is basically the same, but it requires quite a bit of you as a leader. First thing first. If you are a manager who has a strong opinion on any of these, and you are ready to unilaterally decide because you think it’s the right opinion, don’t. Your job here isn’t to make a decision. Your job is to manage the process so no one cares about the eventual decision. I have never come into a team having religious arguments that is healthy otherwise. Religious arguments are to me a symptom of underlying dysfunction. You must understand the underlying dysfunction. That usually includes poor code quality, little code reuse, poor reviews, or inadequate automated testing. Usually there are also other process and management issues. After all, bad codebases aren’t created in a vacuum. Passion people bring to religious arguments can be rerouted into more constructive pursuits. Often the passion is around the religious arguments because that’s all they feel empowered to have an opinion about. Now that you understand the...

Read more


Comparing assemble-your-own vs. kitchen-sink frameworks

In my previous post, I wrote about some of the “performance” tradeoffs in build-yourself vs. off-the-shelf approaches as an analogue for remote vs. collocated teams. I’d like to wade deeper into the build-yourself vs. off-the-shelf approaches, in this case going with an “assemble-your-own” framework (Sinatra with Sequel, Flask with SQLAlchemy) vs. “kitchen-sink” framework (Rails, Django). My goal is not to compare the pros and cons of each, but rather to explain why such arguments are missing the point. I have spent most of my career in video games. Large video game studios often build (built?) nearly everything from scratch. Everything from audio middleware to the game engine itself is often modified at the source code level. Most systems are written in C++, which is difficult to make granular libraries for. All code has strict performance ramifications, so large, critical-path systems that could be generic are often written from scratch. Oftentimes there are proprietary or visual scripting languages for much of the gameplay. There is very little that is actually usable truly off-the-shelf. The opposite is true in web development. Interfaces like Ruby’s Rack or Python’s WSGI provide an extensible way to add application behaviors. Granular libraries are often interoperable enough because the languages have relatively rich datatypes. Because performance is dominated by IO, we rarely care much about the performance of the libraries we depend on. And of course, we have the order of magnitude productivity speedup: we’re using a garbage-collected language. Having this perspective, I have always seen debates about building a product with a kitchen-sink framework like Ruby on Rails compared to an assemble-your-own framework like Sinatra with Sequel as sort of missing the point. We’re incredibly productive in either one! Let’s not argue over scraps. When I look at the entire value stream for a feature/product...

Read more


Performance of remote vs. collocated teams

I saw a tweet the other week about remote teams having lower performance than colocated teams: It's not possible to not take a significant performance hit if you're not collocated. Remote if you must, but don't imagine that it's not costing both money and time to do that. — Allen Holub (@allenholub) January 31, 2018 I thought this spectrum view of “performance” was curious and wanted to explore it more. I have not worked in a fully remote team, so I can’t provide my own anecdotal opinion, but I do have an analogue I feel quite comfortable with: “build yourself” versus “off the shelf.” I’ve been at both ends of the spectrum and observed some common behaviors. Build-yourself engineering teams are more likely to carefully consider what they’re building, and to develop a stronger generic technical expertise. Off-the-shelf teams are more likely to ship quickly and experiment, and focus more on the product.† This is to say that, along one measure, build-yourself teams are “lower performance” than off-the-shelf teams. But success in business is not about the speed of engineers. It is about a whole host of factors. Off-the-shelf teams will ship more product features more quickly. If you are building an enterprise product where you absolutely need to check boxes for sales, this is probably important. If you are shipping a consumer product where you prioritize a clean experience, this is less important. Build-yourself teams will have a steep learning curve for new hires. If you need to keep your team small and are focusing on long tenure, this is not a big deal. If you are planning to grow your team quickly, this can overwhelm the existing team. My point here is that, I don’t consider off-the-shelf or build-yourself inherently worse.‡ I wouldn’t call either one “lower...

Read more


Modern computing is fast

There was a bug I fixed that I was reminded of recently. It’s was small bug – a one line fix – and it didn’t take too long to track down, but it left an impression as to how fast modern computing is. We had some code in an (old) request handler for when a user updates their profile, like this: user.profile.update( profile_fields ) user.update( user_fields ) We had some other code in another part of the system (asynchronous jobs) that listened for profile updates, and would send an email: on 'profile.update' do send_email( 'profile_update', user: user, profile: user.profile ) end We upgraded our RabbitMQ boxes to be more powerful, and then intermittently the email would have updated profile information but outdated user information. You may be able to intuit the bug even with this totally dumbed down example. The asynchronous job ‘profile.update’ event handler was running between the two update calls. There are 4 virtual machines involved here, so I’ll try to illustrate what each of them is doing, and where the race condition existed. Request Handler (Ruby) Postgres Rabbit Async Job (Ruby) Generate profile row update SQL Send profile row update SQL -> -> Write changes to profile row <- Send reply Receive reply <- Publish profile update event -> -> Write event <- Send receipt Receive receipt <- Generate user row update SQL Send event for consumption -> -> Parse and route event Generate user query SQL <- Send user row query SQL Query user row <- Send reply -> -> Receive reply Send user row update SQL -> -> Write changes to user row The race condition is that the Async Job code was running before the user.update running on the Request Handler was committed to the database. So the Async Job code had an...

Read more


How to turn a team around in one easy step

My last four roles have all been “turn-around” scenarios. At CCP Games I had three: as Technical Art Director, I turned around a group that was struggling on executing basic tooling; as Technical Director I had to turn an entire engineering department that was writing an order of magnitude more lines of code than they should have been; as a Senior Engineer in Atlanta I was to figure out how to ship a game that had been in limbo for 8 years. As the head of engineering at Cozy, I needed to figure out how to rebuild the team and product and grow the business 100 times over without increasing costs. I feel like I have a pretty broad view of the turn-around scenario at a variety of levels of leadership (frontline manager to executive), group size (5-100+), and scope (homogenous team to entire company). Over the past weeks, I’ve been reflecting on the commonalities between all these turn-around cases, and I think I found it: Decide what you value, and refuse to compromise on those values. “Legacy” environments come with a lot of baggage and differing views. My job was not to compromise between these views. My job was to create an environment where people could do their best work, which (to me) required establishing values and rejecting compromises that did not advance those values. Reframing compromise in this way meant that solutions came through a shared understanding based on a shared set of values, not some midway point between what two parties wanted. Folks who had a shared understanding of values, no matter how different their point of view or perspective, came away from discussions happier and with better solutions. Here are various values I’ve held over the years, what I see as their obvious implications, and...

Read more


The most important piece of advice I’ve ever gotten

In May 2010, I had been at BioWare for about 2 years, and was ready to quit. Actually, I had quit. My Art Director at the time set up a meeting between myself and Gordon Walton, a the General Manager for BioWare Austin. Despite of being at BioWare for 2 years I had never spoken to Gordon (do you see why I was ready to leave?). Someone told me Gordon likes to hear suggestions and not people whining, so I made sure I came with complaints and solutions. I really loved Austin, I loved the people I worked with, and I didn’t want to leave. But work had become such a struggle, such an unhealthy environment for me, that things needed to change or I needed to go. The necessary change wouldn’t even be so difficult! I wrote down pages of notes and even rehearsed a bit in the mirror. I had good ideas and I wanted Gordon to listen. At worst, he would listen and disagree. At best, he would put me to work fixing some of the very fixable problems. So the day came. I showed up at Gordon’s office. His office was always on the interior; I was told it was because he doesn’t want to seem ‘better’ than other employees with a windowed view in a corner office, though it could be that he disliked sunlight. I sat down. We made some chitchat about my situation, and then he asked me what my problems were. I went on for a few minutes. He listened. When I was done, Gordon gave me some advice that to this day continues to be the most important of my career. Well, Robert, you’re not compatible here and you don’t have a future here. You are a valuable team member...

Read more