Website and Blog of Rob Galanakis: Tools Programmer, Technical Artist/Animator

Recently at work I had the opportunity of spending a day to get Maya and PyCharm working nicely. It took some tweaking but we’re at a point now where we’re totally happy with our Maya/Python integration.

Remote Debugger: Just follow the instructions here: http://tech-artists.org/forum/showthread.php?3153-Setting-up-PyCharm-for-use-with-Maya-2013. We ended up making a Maya menu item that will add the PyCharm helpers to the syspath and start the pydev debugger client (basically, the only in-Maya instructions in that link).

Autocomplete: To get autocomplete working, we just copied the stubs from “C:\Program Files\Autodesk\Maya2013\devkit\other\pymel\extras\completion\py” and put them into all of the subfolders under: “C:\Users\rgalanakis\.PyCharm20\system\python_stubs” (obviously, change your paths to what you need). We don’t use the mayapy.exe because we do mostly pure-python development (more on that below), and we can’t add the stubs folder to the syspath (see below), so this works well enough, even if it’s a little hacky.

Remote Commands: We didn’t set up the ability to control Maya from PyCharm, we prefer not to develop that way. Setting that up (commandPort stuff, PMIP PyCharm plugin, etc.)  was not worth it for us.

Copy/Paste: To get copy/paste from PyCharm into Maya working, we followed the advice here:  http://forum.jetbrains.com/thread/PyCharm-671 Note, we made a minor change to the code in the last post, because we found screenshots weren’t working at all when Maya was running. We early-out the function if “oldMimeData.hasImage()”.

And that was it. We’re very, very happy with the result, and now we’re running PyCharm only (some of us were using WingIDE or Eclipse or other stuff to remote-debug Maya or work with autocomplete).

————————

A note about the way we work- we try to make as much code as possible work standalone (pure python), which means very little of our codebase is in Maya. Furthermore we do TDD, and we extend that to Maya, by having a base TestCase class that sends commands to Maya under the hood (so the tests run in Maya and report their results back to the pure-python host process). So we can do TDD in PyCharm while writing Maya code. It is confusing to describe but trivial to use (inherit from MayaTestCase, write code that executes in Maya, otherwise treat it exactly like regular python). One day I’ll write up how it works in more depth.

It seems the more middleware we use, the more we need to work with temporary files. So for Tech Artists, we usually work with temp files a lot! So it’s important we follow best practices when we need to work with temporary files.

Note: I’ll be using python functions and terminology but this applies to any language.

It goes without saying but, use the tempfile module. Don’t use environment variables or roll something yourself (I’ve seen both done).

Never hard-code the location of a temporary file. I’ve used middleware that does an equivalent of “os.path.join(gettempdir(), ‘myscratch.foo’)“ to get some path. Which means running multiple processes at the same time causes file lock errors! I’ve had to change the temp directory before calling these processes.

Either your directory or your filename should always be programmatically looked up: so “mkstemp(dir=os.path.join(gettempdir(), ‘myscratchfiles’)“ is acceptable, as is “os.path.join(mkdtemp(), ‘myfile.foo’)“.

Try and clean up your files and folders! I say ‘try’ because it isn’t always possible. Sometimes your library may need to return the path to some file it generates. In that case, rather than rely on the caller to clean up the file (which would be a bizarre dependency and docstring!), you’re better off to take in the ‘output path’ to your library, and write the result file there. And your library can clean up the intermediate temp files. That way the caller can be responsible for deleting the result file if it needs (hopefully it doesn’t) because it knows more about the path.

In a lot of cases, I don’t bother cleaning up temp files for internal software. Management of temp files takes a non-negligible amount of work, and if I can keep the software simpler by not worrying about it, I will. I’d rather rely on the developers to keep their hard drives in good shape, which they’ll do anyway.

If you’re running something that will generate a lot of temp files (maybe your test running script), you can also set your temporary directory to a temporary subdir, and then clean that up. Something like:

oldtemp = tempfile.gettempdir()
newtemp = tempfile.tempdir = tempfile.mkdtemp() # Maybe set os.environs TMPDIR/TMP/TEMP as well?
try:
    nose.run() # Or whatever processing you need to do
finally:
    tempfile.tempdir = oldtemp
    shutil.rmtree(newtemp)

And lastly (it’s last because I’m sure people have some religious objection to it), I’ve actually created a custom mktemp function that just calls mkstemp and closes the file descriptor. Just getting a path is useful and worth the performance overhead in (IMO) the vast majority of cases, especially since we often just need the filename to pass to some external process.

What advice do you have for working with temp files (and where is my advice poor)?

In my opinion, good python libraries and frameworks should spend effort guiding you towards the ‘pit of success’, rather than trying to keep you from failing. They do this by spending most effort on things related to the critical path- clear interfaces, simple implementations, thorough documentation.

Which is why singletons are, to me, the worst form of framework masturbation in python. You will never be able to stop people from doing something stupid if they’re determined (in pure python). In the case of a singleton, that means instantiating more than one instance of a type. So spending effort on ‘designing’ singletons is not just a waste of effort, but actively harmful. Just provide a clear way to use a single instance, and your system should fail clearly if it detects an actual problem due to multiple instances (as opposed to, trying to detect multiple instances to keep said problem from happening).

The best method for singletons in python, then, is- whatever is simplest!

1. Some form of module or class state is, to me, the clearest. It requires someone reading or using your code to know nothing more than the most basic python. Just prefix your class def with an underscore, and expose an accessor function to an instance stored on the module (or on the class). The capacity for failure is minimal and the behavior is clear (it requires no behavior modification to the type itself).
2. Overriding __new__ is pretty bad but OK. It requires someone to understand the subtleties of __new__, which is a useful thing to teach someone but, are singletons really the time and place?
3. Using a metaclass is a terrible solution. It has a higher likelihood of failure (how many people understand the nuances of metaclasses!?). Misdirection even for people just reading your code, trying to understand your type’s behavior. Avoid.

A few weeks ago, I held a demo about context managers and decorators. Teaching context managers was straightforward, as expected. Decorators, however, are a much more complex concept. So I used Steve Ferg’s article about the right way to explain python decorators. This was a live demo with little preparation (just wrote down his steps and knew I would write decorators to cache and timeout), to a variety of skill levels, and it was unanimously successful. Everyone came away with a much clearer idea of how decorators work- I saw a lightbulb go off in everyone’s head.

I don’t know why it took so long to figure out how to teach decorators (even harder than first figuring out how to use them!), but Steve’s method works, and I’d encourage you to use it to teach anyone who’s interested.

We’ve all had it- sometimes, under certain conditions, your code errors with an IOError because Windows can’t delete a file. It seems random, and it is. Well, in case you weren’t aware, it’s almost definitely your anti virus software (http://blogs.msdn.com/b/oldnewthing/archive/2012/09/07/10347136.aspx) or at least, some other crazy shit going on.

So a few weeks ago I gave up and said, forget it, I’m just going to write retrying rmtree/remove functions. It took a long time because I refuse to do cargo-cult programming on problems like this, so once I saw that article, I went ahead with it.

I just wanted to spread this information in case you’re in the same situation I was in. You may as well harden and test your little helper retying-delete function I know you have hidden away somewhere, and make it a part of your core libraries.

<No code provided- this is the hacky shit that cannot have a straightforward design and IMO shouldn’t leave your studio, so you’re on your own to figure out an acceptable design and implementation!>

Some people can find me a bit over-earnest in my quest for automation. I’ve finally figured out how to know whether something is worthwhile to automate.

Are you making any decisions when you do this?

And if someone is making decisions that may be unnecessary:

Can we get from A to B without making any decisions?

I value the people I work with and I want them to be more effective. Sometimes people are afraid I’ll automate them out of a job, or they like a manual part of their workflow. More often they take pride in being the manual bottleneck behind a tricky or brittle process.

Humans are valuable because we can make complex decisions very quickly. When I automate an area someone isn’t immediately comfortable with, it’s not because I don’t value what he or she does. On the contrary: because I value their skills and time, I want to see them doing something that requires their unique abilities.

Writing up some performance reviews reminded me of my opinionated fact that a minority of people programming python really understand python. I think in statically typed languages, this is somewhat more acceptable. I think of how interesting “CLR via C#” was. It helped me understand the CLR and IL, as well as some much more general design patterns (damn you, properties!), but had minimal impact for how I used C# (except perhaps regarding performance and GC).

But in a fully dynamic language that has so much power- not understand how the language itself works puts you at a severe disadvantage. I’m not just talking about the one thousand aspects of meta-programming, but even just taking advantage of duck-typing (ie, instead of creating/passing a type, can you just pass a callable that returns a new instance?). Or how decorators work, or how importing works.

I am pretty sure I asked this when I started python but, are there any books (or courses) about python, written for more-or-less experienced programmers (even python programmers), that aren’t about ‘teaching python’ in the normal introductory (or cookbook/reference) way? More along the lines of CLR via C#, or some of the better python guides out there (Idiomatic Python). I am sure there are a large number of people out there who would get huge benefit from such a book or comprehensive guide.

The wife is interested in learning programming so she has been doing the Code Academy JavaScript course. It’s incredible to observe a person learning programming from the ground up, and how their mind works. It’s almost impossible to remember those early weeks of learning how to code yourself.

It’s been fun to see the frustrations, problems, and questions she’s had:

• For loops. So hideous. I don’t know who invented such the syntax “for (i=0; i<100;i++)”, but really I blame the people that continued the tradition into later languages where it is a truly stupid syntax.
• Speaking of i++/i–. What are these things and why do they exist?
• Curly braces. What a waste of space!
• var. When do I use it and when do I not use it?
• Why do I need to end my lines with semicolons?

They also have a python course, I’ll be interested to see how that one goes. I suspect it’ll make less sense since she’ll now expect all languages to work as poorly as Javascript :)

Also to preempt the comments about how this doesn’t mean anything about a language and we shouldn’t design languages for complete programming newbies- yes, I agree. Mostly I found this interesting because the grievances she’s had all have their roots in C-style languages yet they are found in JavaScript which was designed to be novice-friendly. Though admittedly the novice in 1995 and 2012 start with very different dispositions.

I’ll certainly have much more to say once she moves into the rest of JS, and then into Python.

Also, Code Academy is really awesome.

In my line of work I deal with junior programmers constantly. One of the most common non-code mistakes I see them make is using capital letters in their python file names. I’m not sure why it happens so often, to such a diverse crowd, but I guess it comes down to bad examples, and stylistic preference.

Except it isn’t stylistic preference. And unlike my rant about tabs-vs-spaces, it isn’t easy to fix either.

Imagine a simple scenario (note, this only applies to Windows- I should mention in my line of work we are usually on Windows) :

1. You create a file, myModule.py, you put some awesome code in it, and check it into source control (probably Perforce but AFAIK any source control has the same problems).
2. A while later, you rename it to mymodule.py, fix up your code references, and check it in.
3. The code breaks on all your users machines, and you can’t clean it up without manual intervention, or changing your module to be called something else.

Did you see what happened? When you checked in the code, everyone synced down ‘myModule.py.’ When you changed the capitalization, everyone else’s VCS synced down the new file contents but, because the file was already there, didn’t delete myModule.py. So everyone else got the new code calling ‘import mymodule’ but keeps the old filename ‘myModule.py’. Fixing this problem either requires removing the file from people’s client and force syncing (manually or by mass-deploying a fixup script), or just renaming ‘mymodule’ to something else entirely.

The problem stems from python, which is case-sensitive, relying on the filenames of paths, which rely on Windows, which is case insensitive. Don’t bother blaming anyone, it’s just the way it is.

So the way to deal with it? Always use lowercase, which is exactly what the PEP8 style guide suggests.

We’ve finally gotten our test coverage reports published to an accessible place and I’ve been wondering whether including test files as part of the coverage report is generally preferable to excluding them.

Initially I thought they should be excluded- after all, it’s your ‘real’ code people are generally worried about- but now I’m not so sure. You should probably make sure your test files are actually run and covered, as once you have a large amount of tests that often bend in various ways, it isn’t always a given. And actually that’s just the case  we were in today, we have tests that are still not runnable on our CI server, but the report doesn’t show which ones.

Not to mention having your test source easily browseable in the form of a coverage report, which is easier for test-curious people to browse than a source depot.

I think I’m going to re-include tests in the report, despite it skewing our coverage percentage (favorably :)). But I’m really curious if there’s accepted practice or good reasons either way I haven’t thought of.