1- Uses Thread.Start
2- Show no understanding of CPU-vs-IO bound operations
3- Show no understanding of how a computer manages threads
Take this psuedocode for some widely-used routines I ran into today (actually our custom File.Copy method uses this internally!):
function CopyFiles(fromFilenames, toFilenames): for i = 0 to fromFilenames.Count - 1: System.IO.File.Copy(fromFilenames[i], toFilenames[i]) function FastCopyFiles(fromFilenames, toFilenames): #Bucket filenames into arrays, one for each core bucketedFromFiles = BucketFiles(fromFilenames) bucketedToFiles = BucketFiles(toFilenames) for i = 0 to bucketedFromFiles.Count - 1: Thread.Start(CopyFiles(bucketedFromFiles[i], bucketedToFiles[i])) waitForAllThreadsToFinish() #implemented with some counter system
There are so many things wrong with this. I totally understand the idea- the Thread is waiting during IO, so just new up threads to send more IO, while each thread waits for the IO to complete. Here are the major problems:
- Newing threads are expensive! Each thread requires a 1MB stack and takes a significant amount of time to create and destroy.
- Managing threads is expensive! Each core on your computer can only run 1 thread at a time (basically). There are other programs running on your computer, as well as possibly other threads in your program. Windows allows ‘context switching’, which means a CPU binds to a different thread- which requires unloading and loading a thread’s cache onto the CPU, and a host of other stuff. Creating more threads than you have cores means context switches happen more often. More threads will get created in your program when the CLR detects a thread is blocked and there are things to do, or you request one with Thread.Start.
- Your threads are doing NOTHING! While each thread is waiting for the IO to complete, it is doing absolutely nothing. It is just killing time, and your performance.
Naive programming involves parallelizing a process, but not making it asynchronous. Parallelization (especially custom algorithms, not using the built-in ThreadPool/Threading.Tasks.Parallel.ForEach/PLINQ/etc) is good, but you NEED to be wary of IO-bound operations (or threads that launch a separate process, etc.).
The correct approach here is to basically have a single thread (well, just let the ThreadPool manage the threads) to begin an asynchronous write operation, and Wait for the tasks to finish. The ideal is that a thread gets a ‘BeginWrite’ request, runs to the HDD, drops off the request, then comes back up and does more work (probably running back to the HDD to drop off another request). As the HDD finishes the requests, a thread (the same or different ones) can pick up the notification and run a callback, signal that the original request has finished, etc. So no threads are sitting idle waiting for the HDD- they are running around frantically doing work. Which is fine- what we want to avoid is 1) creating new threads, 2) context switches, and 3) inactive threads while there’s other CPU work to do (which means wasted resources).
I’ll go more into the explanation/example for the proper way to implement that FastFileCopy method in a future post (actually probably after I rewrite the one at work). There are already lots of examples of asynchronous IO so you should be able to figure it out yourself. Which you must do if you want to write multithreaded programs. Because you don’t want to be a smart person doing naive programming.
The last part of the video is where I talk about how tech artists need to be ‘ruthless.’ It certainly caught Bill offguard, and gave me a focal attribute to talk about for the rest of the week. “Ruthlessness” will of course need to be a topic for a future post.
Thanks a lot to Bill for doing all this!
The main point of the presentation is understanding how to get your Tech Art and Tools Engineering teams to work together effectively (and why they aren’t working effectively now). I go over each team’s strengths (have you ever considered how differently TA and Engineering are set up?!), how to turn adversarial relationships into positive structures (how are tools supposed to get made when all three departments are competing for the same people’s time?), and actual technical strategies for working together (defining strong data interfaces and laying boundaries for a common codebase).
Please download it, read it over, and tell me what you think!
This was the largest GDC yet, with something like 18,000 attendees and 600+ speakers.
Our Tech Art Bootcamp was a huge success. Almost filled room from 10am to 6pm on Tuesday. Incredible. The Tech Art Roundtables were packed every day, and the Tech Animator Roundtables seemed to go better than last year.
I think my speech, entitled “Ending the Culture War: Uniting Tech Art and Engineering to Create a Better Pipeline” went incredibly well. Not a repeat of last year’s disastrous presentation.
I’ll get the bootcamp slides up this week.
I’m going to be posting about some of the trends I’ve noticed and challenges we’re facing as a discipline.
Tech-artists.org will be moving into the mobile/social age soon with some much needed site improvements.
If you’re in town and want to meet up, just send me an email! I’ll also be at the tech-artists.org meetup on Tuesday night.
The big problem we had (on Tech Art and Engineering teams other than the Tools team I’m on) is that we had no ‘culture of code reviews.’ Everyone just paid lip service to them- even me, because I was never really allowed to be as forceful with requiring over-the-shoulder reviews before all check-ins as I wanted to be. So I’d do most reviews after-the-fact, which was slow, via the clunky format of email, and would often go unheeded.
Code Collaborator, most of all, provides an automatic culture adjustment, because suddenly code reviews are a formal part of the checkin process. This may be obvious or absurd, depending on your experience. But it took a true formalization of the concept via a concrete (and polished!) tool like Code Collaborator to get us where we need to be regarding code quality and code reviews.
I just wanted to put that out there, as I’m up at midnight doing code reviews…
I think after the first three or four years, it’s pretty cast in concrete whether you’re a good programmer or not. After a few years, you may know more about managing large projects and personalities, but after three or four years, it’s clear what you’re going to be. There’s no one at Microsoft who was just kind of mediocre for a couple of years, and then just out of the blue started optimizing everything in sight.
I agree with a caveat (that may not have existed in 1986 but does now). I think certain people’s minds are better set up to do different types of programming, and it may take a few years for them to find that type of programming- and once they do, they may shine, perform significantly better. Different types of programming could be programming at a different level of abstraction (managed vs. unmanaged code), different areas (database, client, graphics, applications), or practices (OOP vs. FP).
That said, I feel like the caveat only applies to maybe 5-10% of the people that wouldn’t otherwise be good programmers.
I think the quote is important to keep in mind when doing hiring- experience can be a very poor indication of competency, skill, and mentoring ability. If you get a veteran, their potential is known, whereas someone with less than three or four years experience has some of his or her fastest growth ahead. So if you consider it that way, all other things being equal, you may be better off hiring someone with less experience.
Fixing things is easy. You just need to know how things work.
Once things once again spiral out of control, he has these cautionary words:
You have to be careful about what you fix. If you fix the valves in an engine, but the bearings are shot, you’ll get more compression, but the engine will still burn up. If you irrigate a desert, you might empty a sea. It’s a complicated business, fixing things.
I really like these words. Knowing how things work is part of our jobs as engineers. But it can easily get you into trouble- what you really need to know is how things work together. Knowing how a system works, not just at its formal level- we can all look at a diagram of a simple engine and understand the parts- but at the micro, atomic level and the high, contextual, historical level, is vital to truly understanding any complex system.
So the hard part is identifying when you know how things work, and when you know how things work together. It’s worth the extra inspection and discussion and prudence to figure it out, and make sure others figure it out.
Iteration is king. But more than that, you need to really explore- Sid Meier has a design rule called “Double it or cut it by half“:
…the probability of success is often directly related to the number of times a team can turn the crank on the loop of developing an idea, play-testing the results, and then adjusting based on feedback. As the number of times a team can go through this cycle is finite, developers should not waste time with small changes. Instead, when making gameplay adjustments, developers should aim for significant changes that will provoke a tangible response.
If a unit seems too weak, don’t lower its cost by 5%; instead, double its strength. If players feel overwhelmed by too many upgrades, try removing half of them… The point is not that the new values are likely to be correct – the goal is to stake out more design territory with each successive iteration.
Imagine the design space of a new game to be an undiscovered world. The designers may have a vague notion of what exists beyond the horizon, but without experimentation and testing, these assumptions remain purely theoretically. Thus, each radical change opens up a new piece of land for the team to consider before settling down for the final product.
-From Soren Johnson’s blog post, originally read in Game Developer Magazine
Especially in tools, we often don’t realize our iterations are often just polish for something that doesn’t work in the first place. You really need to be able to make big, sweeping changes, especially early on, when creating or refactoring a tool. The only way to do this is to make sure your data model is solid and complete, so you can rewrite the entire tool in those two weeks and break as little as possible. This is the most effective way to discover new workflows and make those really big, order-of-magnitude improvements.
But much harder is being allowed to make those changes in the first place- either because the data model is so brittle, or management is averse to risk. The only way to make those sweeping changes, then, is to take risks, but always deliver on your word. Eventually you’ll earn enough trust to be allowed to do them, and enough skill to do them well enough that you can fix that broken data model at the same time.