Blog of Rob Galanakis (@robgalanakis)

Posts by: Rob Galanakis


Could a random hire thrive in your organization?

I made a couple posts (applicant-designed hiring, randomized hiring) about how less-controlled hiring processes could lead to designing an organization where more folks could thrive. It’s largely a thought experiment, so I’ll share my thoughts :) What would need to work for a random hire to thrive? Smooth onboarding and documentation. Nothing can fix getting off on the right foot, so onboarding, technical and otherwise, needs to be in good shape. Decent tooling. If doing anything requires expertise with a bunch of cloud services and debugging tooling that is constantly inefficient, it’s unlikely a random hire would be able to participate. Good test coverage and code quality. Give folks the best change possible of being able to contribute to the position they’re hired into. Robust management practices. Without “culture fit” interviews, you’re likely to get someone outside your norm. You’ll need an adaptive and coherent management strategy. ] Continuous learning. You can’t select for a specific set of skills as easily, so you’ll need to make sure folks can pick up skills on the job. Dealing with poor fits. You can’t depend on your hiring process to prevent them (not that it does now). You’ll need to figure out a compassionate way to part with bad hires. The list goes on and on but these were at front of mind. And wow, these things look like they’d benefit any team, regardless of hiring practices! It seems like most folks would be able to thrive if you had these sorts of things solved. In some ways, this is similar to “what is preventing us from auto-deploying when our tests pass?” Solving that is just a huge win, even if you don’t auto-deploy. Creating an environment where a random hire would likely succeed is much more firmly in your control than...

Read more


Why do we always think our team is so great?

If you ask someone experienced what they think of their team, they’ll usually tell you their team is the best team they’ve ever worked on. It’s so rare to hear someone say they think their team is bad or even mediocre. Why is this? We rationally understand that most teams and employees must be average, near the middle of the bell curve of ability. But we don’t experience it that way. I don’t believe there is some “dark matter” of teams that fill the back of the curve, and I and everyone I know works at the front (and anyway, you can apply the curve to subpopulations). There’s also the factor of, what do we mean by “best team”? We have no single or compound measure of performance. And contexts change: business performance, market forces, type of work, and especially our own mental state and abilities. I suspect the reason we feel we are always on the best team of our career is exactly because we and our teams are so often average, and we can’t consistently measure team ability. In fact, I suspect most of us would believe a statement like “I was on a 70th percentile team” is absurd. But what percentile rank can we assess? I don’t know. Not only can we not measure, we have tremendous psychological reason to believe our current team is exceptional, both for our sake and our coworkers. But despite the inability to measure, effective and ineffective teams do exist. Something like 10% of us are each on truly great or truly awful teams. The 10% on awful teams likely know they are on an awful team (by virtue of thinking their team is awful). But the 10% on great teams can’t likely know they are, given how the other 80%...

Read more


And what if we hired randomly?

A couple posts ago, I wrote about letting candidates choose their own hiring process. But for years, I’ve been toying with the idea of something even more radical — what if we hired randomly? This sounds silly, but wait! Think of a time you’ve transferred teams, or came in to lead an existing team, or had someone from a different group transferred to your team. This situation is mostly random, especially if the “process” they were hired with varied differently from whatever you have now. I am not suggesting you start hiring randomly; what I’m suggesting is that you think about how much randomness is already a factor. It’s unlikely that the greatest teams you’ve worked on were hired with a process you also designed or vetted; or that you even had input into their hiring. But somehow the team became great! Not due to your hiring process. Hmmmmm…. So the idea of “hiring randomly” is about, “how can we put systems into place that a person hired randomly has a better-than-random shot at thriving?” Or, how can we build a team where adding a random person is likely to make it better? In an effort to be less prescriptive, I’m going to leave it there for now, but I may pick it up in the future and I hope it is an interesting thought exercise at least.

Read more


Managers learn lessons on the backs of their reports

When you make mistakes as a manager, you usually don’t pay the price. Your reports are always the ones walking away worse off — they have to deal with the repercussions of your mistake, and probably didn’t learn anything useful. This is an inherent part of hierarchical power dynamics. You can’t wish it away. You can’t smile it away. This is why being an effective manager requires large amounts of compassion. I don’t think it’s bad to feel genuinely upset about the mistakes you make that lead to folks suffering — they always have it worse. Your compassion should lead you to mitigate the effect of your own mistakes on reports who are always going to be more vulnerable. That may skip level and diagonal 1-on-1s, a generous severance, or tons of other tools. And you’ll be making a lot of mistakes, because it’s how we learn. So getting good at handling the inevitable problems you are going to cause for employees is a worthwhile skill to cultivate.

Read more


What if candidates controlled the interview process?

The software engineering interview process at most jobs is similar- phone screen, some programming work, a bit of architecture or systems design, cross-functional (aka “culture fit”) interviews. Companies are all over the place on how well they execute this formula, but the general formula is the same. Not only is the formula consistent, but it’s even more common that there is “the hiring process” (singular), sometimes varied by the specifics of the role or title, even if “the hiring process” is an inconsistent dumpster fire. This is strange to me — there’s compelling research (and at this point hopefully general acknowledgement) that we’re really bad at predicting success of candidates. But we seem to all be marching down this path of having a singular process. As if success is just around the corner. My most famous solution is to hire randomly, but that’s for a future discussion. Short of that — what if we allowed candidates to design their own process? Make it a “choose your own adventure” type of experience. They can choose from a menu of sessions, topics, and interviewers. Let the candidate design a process that shows their experience in the light they want; let them show you their strengths; let them be comfortable and confident. What if we had a process designed for cultivation and strengths, rather than one designed around weeding and finding weakness? It would select the people who could truly thrive and add to the whole, rather than selecting for those who could just survive in the sterile, synthetic world of the interview.

Read more


Reflections on greenfielding MilkRun

When I joined MilkRun in June 2018, the technical situation was… not good. Seeing as I couldn’t even sign up due to errors, I had expected issues. But when I saw a timestamp primary key field which was also used as a foreign key, and other WTFs, I knew I was going to rewrite this thing from the ground up. This was the first project I had truly greenfielded. I have built a lot of new services and repos but there was always other stuff it needed to hook into, and some set of expectation that new code should be similar to the old. This was all fresh, though! So I figured out a set of ideas/principles/strategies I’d develop with, based on what I think worked and didn’t work at previous jobs (and what I was technically competent at), and stuck to it for the next year or so. I enjoy exercises like this because it’s also how I developed most of my management beliefs: researching, experimenting, iterating, then trying it out on a larger scale. I went into Cozy knowing exactly what I was going to try from a management/leadership/cultural perspective. And I went into MilkRun’s rewrite knowing just what I’d do from a technical level. Since retrospective posts are fun and easy to write, I’m going to break down most of these topics into future posts. There are a handful of bigger “strategies” that I had: React Single Page App frontend. I really only knew KnockoutJS from Cozy, which I was not going to use again. Honestly I don’t much like SPAs in theory but I’d rather program them than dynamically rendered apps, so I went with it. Ruby API using Grape, Sequel, and Postgres. I was very happy with this at Cozy and saw no reason...

Read more


Don’t bother with migrations for performance-based indices

I used to write database migrations for performance-based indices, like adding an index to a foreign key column. I don’t do it anymore, and do not recommend the practice. Instead, write your SQL against a production-like database (same schema and data), and run it against production directly as part of your “operations” rather than as part of your development workflow. Caveats: This doesn’t apply to migrations part of the initial schema (may as well do that at once), and it doesn’t apply to indices part of a constraint, like a unique constraint. Also, you may have totally valid special-case exceptions, I’m only speaking as a rule of thumb on project sizes that you’d take advice on my blog from, like from 1-100 devs. When you get past a certain size, you probably already have standards for all these things, so you do you. Writing formal migrations for performance-based indices doesn’t really have any upside. You don’t need them in non-production-like environments because they only matter when there is enough data and traffic to be worthwhile in the first place. They do not change the correctness of the code so are not relevant to the code. They are impossible to develop locally. You need to develop and test them in a production-like environment anyway. They’ll come along with database restores anyway, so your production-like staging or wherever that is based on production data will still get them. The process I see everyone follow ends up being: Run SQL in a prod-like environment to find something that works Copy it to code Go through code review, merging, and deployment CD runs migration on production Verify performance has changed It seems like you can have a much more iterative and faster process to avoid the migration and CD hoops. That said, you...

Read more


Reflections on over-engineering: Cozy

The Good We were processing over $100m/mo in rent at Cozy, nearly all of that on the 1st of the month. We were able to do that in a Ruby monolith by maintaining a disciplined programming team, with thorough automated tests and monitoring. Every time we had a performance issue, we fixed it, whether it was adding indices, redesigning queries, or building auxiliary systems like for better parallelization. I consider Cozy’s payment system an example of appropriately-complex engineering. It wasn’t under-engineered then glued together with hacks, it wasn’t over-engineered and then had a bunch of stuff that didn’t work. We were diligent and rigorous. We carried this discipline to all parts of Cozy’s code base and it paid off. The payments system was necessarily complex from a logic perspective, but not a technical one. I believe it could have scaled another order of magnitude in volume without any sort of technical redesign. The work done on the payments system was not spent wrestling with complexity that did not reflect the innate complexity of the problem being solved. The Bad On the other hand, Cozy was running its monolith directly on AWS. This required one full-time Operations (DevOps) Engineer at a minimum, and eventually a second for redundancy. We easily could have run our entire setup on Heroku, which would have given us better tooling and more features. I would have done this, too, but it turns out the value of our first Ops Engineer was immense outside of Ops (he is the first to point out this “host it in Heroku” situation btw). The problem Cozy was solving had no essential technical complexity, and thankfully we avoided much of it. We ended up over-spending in Ops but it did not infect the rest of the organization with workaround complexity.

Read more


Reflections on over-engineering

My entire career has been spent replacing other people’s applications. The closest I’ve come to greenfielding was spending 3 months rebuilding MilkRun and then migrating data from the old system to the new (none of the code or schema was kept). I think I’ve been pretty successful at replacing other folks’ code and it has given me a particular perspective on the state of our craft. I would wager that poor programming practices have created a fundamental misunderstanding of the complexity of what we do as programmers, which has created all sorts of negative knock-on effects throughout out companies and even the economy. There are, to be sure, some problems that are not easy to solve by their very essence, namely dealing with truly enormous amounts of data/bandwidth/processing. That is, things that are more or less dealing with physical constraints. I don’t suppose that I’d know what works at Cloudflare, or Stripe. But I do know that the majority of startups are not dealing with these sorts of problems (certainly none I’ve ever interviewed or worked at). Of those that are, the complexity is normally limited to a particular team or space. In the next couple posts, I’m going to give examples that I’m comfortable giving from Cozy and Cloudability — this is not an extensive list though and the situation is the rule rather than exception. If these posts do not line up with your experience, I’d be really interested in hearing counter-examples.

Read more


“Professional services” for pre-revenue startups

Something I’m beginning to explore is what a services business providing programming and long-term technical guidance and mentorships for startups would look like. A business that takes the things I know I am good at — mentorship, programming, product management — and provide it to companies who need it in a way that leaves them set to grow and scale. How would this work? My ideal setup would be to find a founder/cofounder who is a relatively inexperienced software engineer, and help them get their business off on the right foot. Showing them how to set up a sustainable platform for future development and prioritize so they can avoid hiring a big team with diminishing returns. This may also mean connecting founders needing “technical cofounders” to candidates, with this business supporting both the business and potential cofounder to make sure they’re a fit and the business transitions. A more straightforward setup would be just building what needs to be built (in which case the payment model gets more interesting), with ongoing support and development as needed. I think this could be a great fit for folks that plan to more slowly grow their business and don’t really need to bring development in-house, or pay 3rd party contractors. I could see in the future, once this business has a staff, that they could potentially transition into actual cofounders of the businesses they are partnering with and leave the service. One other thing I’m planning is to create a couple of open source codebases patterned on what I built for MilkRun (based on what we built at Cozy) to stamp out new projects. I’m really happy with both the patterns and development experience at this point — I considered using something like Rails but I just don’t enjoy programming in it....

Read more