Holiday (product) shipping

This was an interesting holiday season, work-wise, for three reasons.

First: My work was closed down from Dec 20th to Jan 4th (except for Customer Support and whichever developer was on firefighting duty, though that is all remote). We shipped two large products on December 17th, which was a bit too close for comfort, but things went OK and it gave us a few days to fix issues.

Second: I was working a couple hours a day while my son napped. I have quite a backlog of pull requests waiting to get in.

Third: On December 31st at about 5pm, we realized our emails hadn’t been going out. Our email service decided to ship 43,000 lines of code the day before, which resulted in a partial outage for some customers (they sent us success responses but things then broke internally).

What lessons did I learn?

First, if you’re going to ship two days before vacation, make sure your work is solid. We had one deployment on Sunday the 21st for some bugs we didn’t want to live with for 2 weeks, but other than that no new work has gone out. We shipped some solid code, thankfully.

Second, if you’re going to work over a holiday, don’t generate work for others. I really want to get the work I’ve been doing out to production, which would require 1) a code review and 2) a deploy of new code. Even if I skipped code review and deployed myself, if shit hit the fan or I introduced some new bug, I’m making work for others. I took a lot of discipline but I’m proud to say that I have fifteen open pull requests and not a single one is reviewed yet. It’ll be a busy Monday and Tuesday but that’s better than messing with peoples’ vacation.

Third, two weeks is a really long time to shut down. In some ways, shutting down is great, as I’ve written about before. But it sucks not having a good way to get fixes and improvements out to customers. There are a lot of considerations here. I’m not sure what we’ll do next year. It’ll largely be up to the team.

Fourth, you should never, ever ship something directly before a holiday or before you go on vacation. It’s immature and unacceptable. You not only screw over your team when something goes wrong, you screw over everyone depending on your product. They need to jump into action and figure out what’s going on, how to mitigate things, respond to customer complaints, etc. I cannot believe I need to tell anyone this. Don’t ship directly before a holiday.

Anyway, just some thoughts. Happy New Year!