Holiday (product) shipping

by Rob Galanakis on 2/01/2015

This was an interesting holiday season, work-wise, for three reasons.

First: My work was closed down from Dec 20th to Jan 4th (except for Customer Support and whichever developer was on firefighting duty, though that is all remote). We shipped two large products on December 17th, which was a bit too close for comfort, but things went OK and it gave us a few days to fix issues.

Second: I was working a couple hours a day while my son napped. I have quite a backlog of pull requests waiting to get in.

Third: On December 31st at about 5pm, we realized our emails hadn’t been going out. Our email service decided to ship 43,000 lines of code the day before, which resulted in a partial outage for some customers (they sent us success responses but things then broke internally).

What lessons did I learn?

First, if you’re going to ship two days before vacation, make sure your work is solid. We had one deployment on Sunday the 21st for some bugs we didn’t want to live with for 2 weeks, but other than that no new work has gone out. We shipped some solid code, thankfully.

Second, if you’re going to work over a holiday, don’t generate work for others. I really want to get the work I’ve been doing out to production, which would require 1) a code review and 2) a deploy of new code. Even if I skipped code review and deployed myself, if shit hit the fan or I introduced some new bug, I’m making work for others. I took a lot of discipline but I’m proud to say that I have fifteen open pull requests and not a single one is reviewed yet. It’ll be a busy Monday and Tuesday but that’s better than messing with peoples’ vacation.

Third, two weeks is a really long time to shut down. In some ways, shutting down is great, as I’ve written about before. But it sucks not having a good way to get fixes and improvements out to customers. There are a lot of considerations here. I’m not sure what we’ll do next year. It’ll largely be up to the team.

Fourth, you should never, ever ship something directly before a holiday or before you go on vacation. It’s immature and unacceptable. You not only screw over your team when something goes wrong, you screw over everyone depending on your product. They need to jump into action and figure out what’s going on, how to mitigate things, respond to customer complaints, etc. I cannot believe I need to tell anyone this. Don’t ship directly before a holiday.

Anyway, just some thoughts. Happy New Year!

No Comments

More effective interviews

by Rob Galanakis on 30/12/2014

David Smith over at baleful.net makes some interesting points about the length of most interviews:

So mathematically, you will most likely get the highest confidence interval with: 1) Resume screen, 2) Phone interview, 3) In-person interviews 1-3. From the above, this should represent about 50% of the total causes, but should produce 91% of the total effect. Adding additional interview steps after that 91% brings only incremental improvement at best and backslide at worst.

He makes an extremely compelling argument, and I encourage you to read the entire piece. That said, I still prefer a full day of interviews as both the interviewer and interviewee.

The interviewee angle is easy. I enjoy interviews. I like to dig into my potential employer. I want to grill your second-string players. I want to hear how junior people feel treated. I want as much information as possible before making my choice. But I know this is just me, and people who are less comfortable with interviews probably prefer shorter ones. I also admit I don’t think I’ve learned anything in the second half of a day of interviewing that would have made me turn down a job. But I have learned things that helped me in my job once hired.

The benefits of full-day interviews for the interviewers is much more complex. There are several factors:

  • We have diverse backgrounds and expertise, and each group brings a unique perspective. Candidate postmortems are not dominated by the same couple interviewers.
  • I want to give as many people experience interviewing as possible. I consider it an important skill. Limiting things to three in-person interviews means the interviewers are all “musts” and I don’t get to experiment at the periphery with groups or combinations.
  • People want to be a part of the process. I’ve personally felt frustrated when left out of the process, and I know I’ve frustrated others when I’ve left them out.

For a developer role, I want them to meet with at least: founders, ops, lead developer, two developers, myself. We’re at an absolute minimum of 7. That is with a narrow set of views, without inexperienced interviewers, and leaving good people out. What am I supposed to do?

  • For starters, the interview process should be more transparent and collaborative. Ask the interviewer if they want a full day, two half days, morning or afternoon, etc.
  • No group lunches. I’ve never gotten useful feedback from a group lunch. Keep it down to one or two people. A candidate just doesn’t want to embarrass themselves, so they just shut up, and side conversations dominate.
  • Avoid solo interviews. I used to hope to solo interview everyone. But over time, I’ve found that pairing on interviews enhances the benefits listed above. There are still times I will want a solo interview, but in general I will pair.
  • Cut the crap. Interviewers should state their name and role. Don’t bother with your history unless asked. Don’t ask questions that are answered by a resume. Instead of “tell us about yourself” how about “tell us what you’re looking for”.
  • Keep a schedule. Some people are very bad at managing time. If someone isn’t done, too bad, keep things moving. They will eventually learn how to keep interviews to their allotted time.

Thanks to David for the insightful post. I’ll continue to keep full-day interviews, but we’ll definitely change some things up.

6 Comments

IKEA instructions are the best

by Rob Galanakis on 27/12/2014

I don’t say it with a hint of sarcasm. I’ve put together a lot of furniture lately, and IKEA instructions are the only instructions that are consistently correct and unambiguous. In dozens of units, I’ve confirmed one case of an ambiguous step. But even in that case, I was able to read ahead and eliminate the ambiguity.

Compare this to almost every other piece of furniture I’ve put together. The drawings are often ambiguous, and even worse, the furniture can be constructed in multiple ways. This is rare with IKEA furniture. You may get to the end and find out you messed up, but things won’t really fit together. With my son’s crib, though, I had moulding pointing the wrong way with no structural effects. Unacceptable.

Assembling furniture from basic components is necessarily complicated. IKEA does a great job embracing this complexity by supplying extremely concise-yet-precise instructions and products where the construction process is considered in the design. My guess is that most people who have problems with IKEA construction jump in without understanding what they are doing. Fiberboard planks and screws are deceptively simple.

I think of this lesson often with the design of complex systems. Anything that deals with ACH (credits and debits in the US) is necessarily complex. You can only abstract to a certain level. Did you know that an ACH payment can transition from Succeeded to Failed? Attempting to “hide” the complexity of ACH, like we successfully hide the complexity of a file system, is a fool’s errand. Instead of making a payments API that’s simple to use, it’s be much better to make one that’s precisely defined, thoroughly tested, and well documented. There are still some problems that require a little bit of RTFM. It’s better to make this complexity front and center in a design like IKEA furniture, than to gloss over it and end up with client code that is built like second-rate DIY furniture.

1 Comment

Free Practical Maya Programming with Python eBooks

by Rob Galanakis on 23/12/2014

Merry Christmas and happy holidays everyone,

Last week I asked my publisher if I could make the Practical Maya Programming with Python eBook totally free. I was told some good news and bad news.

The bad news is, they won’t make it free. The good news is, my editor said that Packt often runs free eBook campaigns, and would make the book part of the free campaign whenever they come up. I will blog here when they do (and also please tweet me @techartistsorg if I miss it).

If you can acquire a pirated copy of my book, I encourage you to do so. Packt does not use DRM as far as I know, so just ask a friend who has the book.

Sorry I can’t make it totally free right now, as much as I want to. It sucks to not have full control over something you have personally invested so much in, but I don’t have the energy to fight my publisher on this one (and the fact that they’re DRM-free makes this much less of an issue).

Enjoy, and please leave me a review on Amazon.

2 Comments

We’re not so different, you and I

by Rob Galanakis on 21/12/2014

Ben Sandofsky wrote a post about why QA departments are still necessary, specifically with regards to mobile app development. He makes a good point: mobile apps create a distribution bottleneck that makes very rapid iteration impossible. I agree, and this is a good angle to think about. I would have been happy with an article focused on this.

Ben is clearly a talented guy but this post was insane. In a literal sense. It is a rant for anti-Agile curmudgeons at best, and would leave me questioning the experiences of anyone that thinks this way at worst.

Websites ship embarrassing bugs all the time. They get away with it because they didn’t ship it to all users. You roll-out everything out to 1% of users, and watch your graphs. If things look good, slowly roll out to 100%.

The idea that this is this sort of incremental rollout is ubiquitous amongst web developers is crazy. It requires infrastructure, code designed to support split testing, experienced operations engineers, robust monitoring, a disciplined process, and more. The institutions with this sort of sophistication all have strong automated testing environments. Which brings me to my next issue:

I think automated testing accelerates development, but I haven’t seen a direct correlation between testing and quality. On projects with massive, high quality test coverage, I’ve seen just as many bugs slip through as projects with zero coverage.

This is the software equivalent to climate change denial. Where does this experience come from? I am not sure I’d be able to find a single developer who would corroborate this. Oh, right:

Tell a game developer you don’t need [QA], they’ll tell you you’re nuts.

The game industry is full of these folks who believe what they are doing is such an untestable snowflake. Unsurprisingly, games have historically been the buggiest software around. Never, ever look at game development as an example of how to do QA right. Not just automated testing, but manual QA too.

…a great QA team is far from a bunch of monkeys clicking buttons all day.

Game development has a hallmark technique of hiring masses of QA people and have massive layoffs at the end of projects. There is an entire website dedicated to tales of horror from QA people. It makes The Daily WTF look like paradise.

Take the unicorn of “two week release cycles.” As you build infrastructure for faster releases, simple code becomes unwieldy. Tasks that should take hours take weeks.

What does this even mean? There are endless apps on two week release cycles. I am confused how building infrastructure for faster iterations ends up adding complexity to simple code or tasks.

Disciplined development is a lost art.

You could make this argument when we moved away from punch cards. But the idea that success in mobile apps is achieved through discipline, but success on the web can be achieved by recklessness, is beyond baseless. It’s downright insulting.

I consider it a tragedy that, when faced with the reality of App Store distribution bottlenecks, Ben’s answer is to go back to the process of yesteryear and throw out the lessons we’ve learned. Why not invent new ways of building in quality? New ways of iterating on apps faster? There are so many interesting problems to solve.

Finally, Ben cautions:

Today, any web developer who wants to stay employed has learned to build apps. If web companies want to remain relevant, they’ll have to do the same.

I have a better warning. Don’t throw away the incredible advances we’ve made over the last decade. Don’t downplay the success and rate of innovation in web development as something that doesn’t apply. Don’t throw away the universal “good idea-edness” of automated testing. Don’t rely on a separate department to enforce quality. Don’t stop looking for ways to make development better.

1 Comment

Grabbing for good enough

by Rob Galanakis on 15/12/2014

Uncle Bob, who I consider my favorite programming writer, had a post a few weeks ago titled “Thorns around the Gold“. In it he describes how writing tests for your core functionality first can be harmful. Instead, Uncle Bob prefers to probe for “thorns” around the “gold” first.

I shy away from any tests that are close to the core functionality until I have completely surrounded the problem with passing tests that describe everything but the core functionality. Then, and only then, do I go get The Gold.

I haven’t been doing TDD for nearly as long as Uncle Bob but I was shocked to read this. I’ve always learned and taught that you should create positive tests first, and only need as many negative tests as you feel are warranted. While you may not grab the gold immediately, you at least step towards the gold. How many thorns you expose is a judgement call. In Python, most people don’t even bother validating for None inputs, and instead just let things raise (or not). Of course, this depends on your users. For libraries limited to one internal application, I wouldn’t “probe many hedges.” For open source libraries, I validate pretty aggressively.

Of particular interest was this:

I often find that if all the ancillary behaviors are in place before I approach the core functionality, then the core functionality is much easier to implement.

I always thought you should only program what you need and no more. It seems very strange to assume the ancillary behaviors will be needed. It seems like a violation of YAGNI.

I have been trying to reconcile Uncle Bob’s advice here, and the TDD best practices I’ve learned and developed. But I cannot. Either I’ve been receiving and giving years of bad advice, or Uncle Bob has made a rare mistake.

2 Comments

Qt Designer is harmful, exhibit A

by Rob Galanakis on 11/12/2014

Last week, iker j. de los mozos posted a Qt tutorial on his blog. The post was retweeted a number of times, so I figure people liked it.

The post exemplifies what is wrong with the Qt Designer, and also how a little more investment in learning can pay dividends for your career.

I know it’s unfair to give people code reviews on things they just put out for free, but I consider it even worse to allow people to continue to use the Qt Designer with a clear conscience. I thank Ike for his post, and for syndicating his feed on Planet Tech Art, and hope that no one takes my writeup below personally. It’s not an attack on a person, it’s trying to point out there there is a much better way to do things.

There are 117 lines of Python code in Ike’s tool for locking and unlocking transformation attributes. This sounds like a small amount, but is for an experienced Pythonista it indicates huge waste. For comparison, the entire ZMQ-based request/reply client and server I built for Practical Maya Programming with Python is the same size or smaller (depending on the version). If we take a closer look at his code (see link above), we can see a ton of copy and pasted functionality. This is technically a separate concern from the use of the Designer, but in my experience the two go hand-in-hand. The duplication inherent in GUI tools carries over to the way you program.

Let’s look at some refactored code where we build the GUI in code (warning, I haven’t tried this code since I don’t have Maya on this machine):

from functools import partial
from PySide import QtGui
import pymel.core as pmc
import qthelpers

OPTIONS = [
    dict(label='Position', btn='POS', attr='translate'),
    dict(label='Rotation', btn='ROT', attr='rotate'),
    dict(label='Scale', btn='SCALE', attr='scale')
]

def setLock(obj, attr, value):
    obj.setAttr(attr, lock=value, keyable=not value)

def isAttrLocked(obj, attr):
    return obj.getAttr(attr, q=True, lock=True)

def toggleRowCallback(attr):
    obj = pmc.ls()[0]
    value = isAttrLocked(obj, attr + 'X')
    for axis in 'XYZ':
        setLock(obj, attr + axis, value)

def toggleCellCallback(attr, state):
    obj = pmc.ls()[0]
    setLock(obj, attr, state)

def makeRow(options):
    return qthelpers.row(
        [QtGui.QLabel(options['label'])] +
        map(lambda axis: qhelpers.checkbox(onCheck=partial(toggleCellCallback, options['attr'] + axis)), 'XYZ') +
        qhelpers.button(label='lock ' + options['btn'], onClick=partial(toggleRowCallback, options['attr']))
    )

def main():
    window = qthelpers.table(map(makeRow, OPTIONS), title='lockAndHide UI', base=QtGui.QMainWindow)
    window.show()

Why is this code better? Well, for starters, it’s less than a third of the size (37 lines) and there’s less duplication. These are very good things. When we want to change behavior- such as auto-updating the checkboxes when our selection changes- we can put it in one place, not nine or more.

So the code is better, but what other benefits are there to not using the Designer?
– We pull common primitives, like a “row” (QWidget with HBoxLayout) and “table” into a qthelpers module, so we can use this across all GUIs. This saves huge amounts of boilerplate over the long run, especially since we can customize what parameters we pass to it (like onClick being a callback).
– The GUI is clear from the code because the UI is built declaratively. I do not even need to load the UI into the Designer or run the code to understand it. I can just read the bottom few lines of the file and know what this looks like.
– You learn new things. We use functools.partial for currying, instead of explicit callbacks. This is more complicated to someone that only knows simple tools, but becomes an indispensable tool as you get more advanced. We are not programming in Turtle Graphics. We are using the wonderful language of Python. We should take advantage of that.

Again, I thank Ike for his blog post, and hope I haven’t upset anyone. Ike’s code is pretty consistent with the type of code I’ve seen from Technical Artists. It’s time to do better. Start by ditching the Designer and focusing on clean code.

5 Comments

The QA Department Mindset

by Rob Galanakis on 8/12/2014

From this post by Rands, titled “The QA Mindset”:

At the current gig, there’s no QA department. […]

My concern is that the absence of QA is the absence of a champion for aspects of software development that everyone agrees are important, but often no one is willing to own. Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective. Traditionally, they are known as the folks who break things, who find bugs, but QA’s role is far more important. It’s not that QA can discover what is wrong, they intimately understand what is right and they unfailingly strive to push the product in that direction.

I believe these are humans you want in the building.

At my current job, we don’t have a QA department either. And like Rands, I wasn’t comfortable at first. I’ve worked on teams without QA, but an entire company without a QA Department? I’ve certainly had questions about the use of a QA department, but does that mean they are a bad idea?

Yes, and this line in Rands’ defense is why:

Unit tests, automation, test plans, bug tracking, and quality metrics. The results of which give QA a unique perspective.

I am a staunch believer of “building quality in.” Every bug that slips out is a failure of your development process. The way to higher quality is not to find, or fix, more bugs. It’s to avoid them in the first place.

If you rely on QA to champion unit testing, automation, bug tracking, and quality metrics, your development process is lacking its most important tools and measures to improving quality. Quality can’t be imposed by QA, it must grow out of enabled and engaged development teams.

I have a saying: “Don’t hire to fix a problem.” If you have a quality problem, hiring a QA department isn’t going to fix it. You instead hide the systematic problems that cause quality issues in the first place.

This is not to say “the QA mindset” isn’t valuable. It is. One of my best hires was Bjorgvin Reynisson, who was a Test Engineer at Nokia and I hired as a QA Engineer at CCP. He was embedded with the graphics engine team and he helped them develop extensive automated correctness and performance testing systems. He worked with them to recognized holes in their process and test coverage. He helped with tracking issues and increasing quality. This is the “QA Mindset” I treasure, and this type of person is invaluable to development teams. Bjorgvin unlocked a latent “culture of quality” in the team he was a part of.

I contrast this “QA Mindset” with the “QA Department Mindset“. The QA Department Mindset has two damning characteristics. First, it is innately adversarial, as Rands notes.

Yes, there is often professional conflict between the teams. Yes, I often had to gather conflicting parties together and explain[…]

Second, it is by definition a separate department, which creates obstacles to better integrating engineering and QA.

Bjorgvin should be spending time with his teammates and the rest of the developers figuring out how to improve the entire development process. He should not be spending time with other QA personnel focused on QA functions. When I was Technical Director for EVE Online, I made sure there were minimal discussions gated by job title. Talk of a trade went on in Communities of Practice, which were open to all. Sometimes this didn’t happen, and those times were mistakes.

Like Rands says:

Yes, we actually have the same goal: rigorously confirming whether or not the product is great.

If that’s the case, QA should not be broken out into a separate department. QA should be working side by side, reporting into the same people, measured by the same success metrics, contributing to the holistic success of an entire product.

I love the QA Mindset. It’s tragic that having a QA Mindset gets confused with having a QA Department.

2 Comments

Long live Slack, down with egotistical email

by Rob Galanakis on 17/11/2014

We use Slack for team communication at Cozy. I struggled with the transition. When I reflected on my struggles, it made me better understand what a destructive format email is for workplace communication.

A quick disclaimer. This is only about work communication and not personal communication. I love email. I think email will be around for a long time and I will lament if and when it goes away. I just don’t think we should be using email for work.

Oration is the highest form of feeding an ego. You craft your message carefully. You research, write, and rehearse. Finally, you take the stage. You command everyone’s attention. And once you’re done, an important topic has been thoroughly addressed and everyone can go on with their lives, better off after hearing what you said.

Email is oratory without the speaking* (or skill). My problems with email stem from when it is used for one-way communication. I suspect that most emails I’ve ever received from anyone in management have been one-way. Generally these emails are meant to, first and foremost, communicate the sender/manager’s self-importance. Often the email contains a nugget of actual information which should be hosted elsewhere. Sometimes the email is an announcement no one understands. And as a rule, you can’t rely on people reading the email you send anyway.

When you craft a long email, like an orator crafts a speech, it is an ego boost. Each one is a masterpiece. You are proud of your fine writing. When you craft a long chat message, on the other hand, you look like a dramatic asshole. It puts in stark perspective how awful the written format is for important or high-bandwidth communication. I’ve never seen someone post a 300-word message to chat. How many 300-word emails do you have in your inbox?

Removing email also levels the playing field for communication. You don’t need to be a manager or orator. Everything you write has a visibility you can’t change. You choose your audience based on topic. Is there a question about a product’s design? Well, it goes into the product or design channel, whether you are Executive Emperor or QA Associate II. Also, no one really wants to read your dramatic flair so please keep it short and to the point.

I used to get frustrated when I’d write an excellent email, send it out, and within a few minutes someone would reply with a message like “Yeah, just to build on what Rob said, it’d be a good idea to do X.” You idiot! You are an Ice Cream Truck driving through the State of the Union. But of course, the problem was mine, playing a manipulative game, focusing too much on this amazing message I’d created. Sometimes these emails would be about the manipulative games people were playing and how we weren’t focused on the employees and customers and things that were actually important.

Email in the workplace is a systematic problem. We take it for granted. We use it constantly. We don’t question it. But email has a cost. It feeds into the already inflated ego of managers. It encourages one-way communication. It is wonderful for grandstanding. We spend a lot of time crafting museum-quality correspondence no one wants to read. And in the end, there are better ways to accomplish what we use it for.


* One of the greatest “speeches” of all time, Pro Milone by Cicero, was written, not spoken. We know great orators by their writing, not their speaking.

No Comments

First, do no harm

by Rob Galanakis on 13/11/2014

From a wonderful post by Matt Williams about the type of business he is looking for:

A Business Manifesto
We are uncovering better ways of running a business and helping others do it.
Through this work we have come to value:
People and interactions over profits and prestige
Quality service over quantity of service
Customer relationships over contract negotiation
Flexibility over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

In a nutshell I want to work for a company which values people — both inside and out of the company. I want to work where people strive to do things right.

When I go home, I want to be able to look in the face of my daughter and not have to make excuses for the work that I do and the effect it has on others.

Sums things up nicely (and definitely what we aspire to at Cozy, by the way we’re hiring).

It is a reason I left the video games industry. I wanted to use my skills to do something I felt was more constructive.

But more than that, I was amazed and frustrated with how the industry was run (almost as bad as films). Mass layoffs even on successful projects. Over-managed projects that go on for 4, 5, 6 years and are cancelled. Creating an exploitative product in order to milk a customer base. Huge budgets, huge marketing, appeals to lowest common denominators (often sexual). There are good companies but the business models are so insane that you can be around for 10 years and fold tomorrow.

2 Comments