Archive of articles classified as' "Tools and Pipeline"

Back home

Building Sphinx documentation for unfriendly code

11/01/2015

Some Twitter friends were discussing how to get Sphinx to work with mayapy to build documentation for code that runs in Autodesk Maya. I’ve had to do this sort of thing extensively, for both Maya and editor/game code, and have even run an in-house Read The Docs server to host everything. I’ve learned a number of important lessons, but most relevant here is:

Always generate your documentation using vanilla Python. Never a custom interpreter.

There’s no philosophical reason for this*. I’ve just found it, by far, the path of least resistance. All you have to do is some mocking in conf.py:

import mock
for mod in ['maya.cmds', 'pymel.core']: # and whatever else you need
    sys.modules[mod] = mock.MagicMock()

(I do not have the code in front of me so this may be slightly wrong. Perhaps an ex-colleague from CCP can check what used to be in our conf.py.)

Now when Sphinx tries to import your module that has import pymel.core as pmc, it will work fine. That is, assuming your modules do not have some nasty side effects or logic on import requiring correctly functioning modules, which you should definitely avoid and is always unnecessary.

When your documentation generation breaks, it’s now a simple matter of adding a string in one place, rather than a several hour debugging session.

Don’t say I didn’t warn you!


* If anything, I’m philosophically more inclined to use mayapy. So that should tell you what sort of bogeymen await!

No Comments

Free Practical Maya Programming with Python eBooks

23/12/2014

Merry Christmas and happy holidays everyone,

Last week I asked my publisher if I could make the Practical Maya Programming with Python eBook totally free. I was told some good news and bad news.

The bad news is, they won’t make it free. The good news is, my editor said that Packt often runs free eBook campaigns, and would make the book part of the free campaign whenever they come up. I will blog here when they do (and also please tweet me @techartistsorg if I miss it).

If you can acquire a pirated copy of my book, I encourage you to do so. Packt does not use DRM as far as I know, so just ask a friend who has the book.

Sorry I can’t make it totally free right now, as much as I want to. It sucks to not have full control over something you have personally invested so much in, but I don’t have the energy to fight my publisher on this one (and the fact that they’re DRM-free makes this much less of an issue).

Enjoy, and please leave me a review on Amazon.

1 Comment

Qt Designer is harmful, exhibit A

11/12/2014

Last week, iker j. de los mozos posted a Qt tutorial on his blog. The post was retweeted a number of times, so I figure people liked it.

The post exemplifies what is wrong with the Qt Designer, and also how a little more investment in learning can pay dividends for your career.

I know it’s unfair to give people code reviews on things they just put out for free, but I consider it even worse to allow people to continue to use the Qt Designer with a clear conscience. I thank Ike for his post, and for syndicating his feed on Planet Tech Art, and hope that no one takes my writeup below personally. It’s not an attack on a person, it’s trying to point out there there is a much better way to do things.

There are 117 lines of Python code in Ike’s tool for locking and unlocking transformation attributes. This sounds like a small amount, but is for an experienced Pythonista it indicates huge waste. For comparison, the entire ZMQ-based request/reply client and server I built for Practical Maya Programming with Python is the same size or smaller (depending on the version). If we take a closer look at his code (see link above), we can see a ton of copy and pasted functionality. This is technically a separate concern from the use of the Designer, but in my experience the two go hand-in-hand. The duplication inherent in GUI tools carries over to the way you program.

Let’s look at some refactored code where we build the GUI in code (warning, I haven’t tried this code since I don’t have Maya on this machine):

from functools import partial
from PySide import QtGui
import pymel.core as pmc
import qthelpers

OPTIONS = [
    dict(label='Position', btn='POS', attr='translate'),
    dict(label='Rotation', btn='ROT', attr='rotate'),
    dict(label='Scale', btn='SCALE', attr='scale')
]

def setLock(obj, attr, value):
    obj.setAttr(attr, lock=value, keyable=not value)

def isAttrLocked(obj, attr):
    return obj.getAttr(attr, q=True, lock=True)

def toggleRowCallback(attr):
    obj = pmc.ls()[0]
    value = isAttrLocked(obj, attr + 'X')
    for axis in 'XYZ':
        setLock(obj, attr + axis, value)

def toggleCellCallback(attr, state):
    obj = pmc.ls()[0]
    setLock(obj, attr, state)

def makeRow(options):
    return qthelpers.row(
        [QtGui.QLabel(options['label'])] +
        map(lambda axis: qhelpers.checkbox(onCheck=partial(toggleCellCallback, options['attr'] + axis)), 'XYZ') +
        qhelpers.button(label='lock ' + options['btn'], onClick=partial(toggleRowCallback, options['attr']))
    )

def main():
    window = qthelpers.table(map(makeRow, OPTIONS), title='lockAndHide UI', base=QtGui.QMainWindow)
    window.show()

Why is this code better? Well, for starters, it’s less than a third of the size (37 lines) and there’s less duplication. These are very good things. When we want to change behavior- such as auto-updating the checkboxes when our selection changes- we can put it in one place, not nine or more.

So the code is better, but what other benefits are there to not using the Designer?
– We pull common primitives, like a “row” (QWidget with HBoxLayout) and “table” into a qthelpers module, so we can use this across all GUIs. This saves huge amounts of boilerplate over the long run, especially since we can customize what parameters we pass to it (like onClick being a callback).
– The GUI is clear from the code because the UI is built declaratively. I do not even need to load the UI into the Designer or run the code to understand it. I can just read the bottom few lines of the file and know what this looks like.
– You learn new things. We use functools.partial for currying, instead of explicit callbacks. This is more complicated to someone that only knows simple tools, but becomes an indispensable tool as you get more advanced. We are not programming in Turtle Graphics. We are using the wonderful language of Python. We should take advantage of that.

Again, I thank Ike for his blog post, and hope I haven’t upset anyone. Ike’s code is pretty consistent with the type of code I’ve seen from Technical Artists. It’s time to do better. Start by ditching the Designer and focusing on clean code.

5 Comments

Behavioral testing is the bee’s knees

10/11/2014

I have been doing Test Driven Development (TDD) with xUnit-based frameworks (like unittest and NUnit) for a number of years now, and started with RSpec in Ruby and Jasmine in JavaScript a few months ago. RSpec and Jasmine are behavioral testing frameworks that facilitate Behavioral Driven Development (BDD). BDD is really no different from “normal” TDD except for the frameworks used. BDD frameworks facilitate a different way of designing tests.

Behavioral testing is awesome and makes xUnit-style testing seem barbaric and uncivilized in comparison. I didn’t see the big deal for a long time. My xUnit style tests served me just fine. But now I’m hooked.

If you are a TDD person (or do any unit testing), I encourage you to try out BDD. You should get the hang of it quickly. It can take a little while to learn BDD idioms, but once you start doing things like custom matchers, it’s absolutely amazing.

In future posts I’ll try to discuss more of the differences between TDD and BDD, but for now I just provide a hearty endorsement of BDD and frameworks like RSpec and Jasmine.

3 Comments

GeoCities and the Qt Designer

25/08/2014

In a review of my book, Practical Maya Programming with Python, reviewer W Boudville suggests my advice of avoiding the Qt Designer is backwards-looking and obsolete, such as writing assembler instead of C for better performance, or using a text file to design a circuit instead of a WYSIWYG editor. I am quite sure he (assuming it is a he) isn’t the only person with such reservations.

Unfortunately, the comparison is not at all fair. Here’s a more relevant allegory:

Hey, did you hear about this awesome thing called geocities? You can build a website by just dragging and dropping stuff, no programming required!

We’ve had WYSIWYG editors for the web for about two decades (or longer?), yet I’ve never run into a professional who works that way. I think WYSIWYG editors are great for people new to GUI programming or a GUI framework, or for mock-ups, but it’s much more effective to do production GUI work through code. Likewise, we’ve had visual programming systems for even longer, but we’ve not seen one that produces a result anyone would consider maintainable. Sure, we’ve had some luck creating state machine tools, but we are nowhere close for the more general purpose logic required in a UI. And even these state machine tools are only really useful when they have custom nodes written in code.

Finally, WYSIWYG editors can be useful in extremely verbose frameworks or languages. I wouldn’t want to use WinForms in C# without the Visual Studio Designer. Fortunately for Pythonistas, PySide and PyQt are not WinForms!

I have no doubt that at some point WYSIWYG editors will become useful for GUI programming. Perhaps it will require 3D displays or massively better libraries. I don’t know. But for today and the foreseeable future, I absolutely discourage the use of the Qt Designer for creating production GUIs with Python.

12 Comments

Deploying a C# app through pip

28/05/2014

“If we bundle up this C# application inside of a Python package archive we can deploy it through our internal CheeseShop server with very little work.”

That sentence got me a lot of WTF’s and resulted in one of the worst abuses of any system I’ve ever committed.

We had a C# service that would run locally and provide an HTTP REST API for huge amounts of data in our database that was difficult to work with.* However, we had no good way to deploy a C# application, but we needed to get this out to users so people could start building tools against it.
I refused to deploy it by running it out of source control. That is how we used to do stuff and it caused endless problems. I also refused bottlenecks like IT deploying new versions through Software Center.

So in an afternoon, I wrote up a Python package for building the C# app, packaging it into a source distribution, and uploading to our Cheese Shop. The package also had functions for starting the packaged C# executable from the installed distribution. The package then became a requirement like anything else and was added to a requirements.txt file that was installed via pip.

What I initially thought was an unsightly hack ended up being perfect for our needs. In fact it caused us to eliminate a number of excessive future plans we had around distribution. I certainly wouldn’t recommend this “technique” for anything more than early internal distribution, but that’s what we needed and that’s what it did so effectively.

Getting something up and running early was extremely useful. It’s important to walk the line between the “we need to do several weeks of work before you see any result” and “I can hack something together we’re going to regret later,” especially for infrastructure and platform work. If code is well-written, tested, documented, and explained, you shouldn’t have qualms about it going into (internal) production. If the hackiness is hidden from clients, it can be easily removed when the time comes.


* Reworking the data was not an option. Creating this service would have allowed us to eventually rework the data, by providing an abstraction, though.

7 Comments

Global Glob

4/04/2014

I am cleaning out my drafts and found this two year old post titled “Globals Glob” with no content. The story is worth telling so here we go.

There was a class the EVE client used to control how missiles behaved. We needed to start using it in our tools for authoring missiles with their effects and audio. The class and module was definitely only designed (I used the term loosely) to run with a full client, and not inside of our tools, which are vanilla Python with a handful of modules available.

My solution was the GlobalsGlob base class, which was just a bag of every singleton or piece of data used by the client that was unavailable to our tools. So instead of:

serviceManager.GetService('someservice').DoThing(self.id)

it’d be:

self.globalsGlob.DoThing(self.id)

The ClientGlobalsGlob called the service, but FakeGlobalsGlob did nothing. The GlobalsGlob allowed us to use the missile code without having to rewrite it. A rewrite was out of the question, as it had just been rewritten, except using the same code. (sigh)

Unsurprisingly, GlobalsGlob was super-fragile. So we added a test to make sure the interface between the client and fake globs were the same, using the inspect module. This helped, but of course things kept breaking.

This all continued until the inevitable and total breakdown of the code. Trying to use the missile code in tools was abandoned (I think it was, I have no idea what state it’s in). This was okay though, as we weren’t using the missile tools much after those few months. GlobalsGlob served its purpose, but I will never be able to decide if it was a success or failure.

2 Comments

Introducing Host The Docs

22/03/2014

A month or so ago I created Host the Docs, and it’s been quietly running at work some success, so I figure it’s time to talk about it. The code is on GitHub at https://github.com/rgalanakis/hostthedocs, and here’s a description from the readme:

Host the Docs is a simple way to host static code documentation. Its main use is as a self-hosted server for your organization’s private documentation. Better alternatives are available for open source projects, such as Read the Docs or Github Pages.

Host the Docs was created after a long day of banging my head against the wall trying to get Read the Docs set up with private GitHub repositories, and having helped develop a plugin to get it to work with Perforce previously. What the world needed was a way to easily host documentation from several projects, from any source or language or SCM.

Seriously, let other people generate their own docs, I just want to Host the Docs!

It’s super easy to get set up, and requires no database. You can even just serve everything through Flask, if you don’t want to set up a webserver like Apache or nginx. It’s pretty well tested and configurable and thoroughly documented. We had it running on a new Linux VM in just a few minutes, and use it to host HTML documentation for Python and C# projects (we used Sphinx to generate the Python docs and Doxygen for C#).

I encourage you to try it out if you are interested in painless documentation hosting! I’m also very open to fixes/changes/pull requests, as long as it doesn’t overcomplicate things.

No Comments

Using Sonar for static analysis of Python code

15/02/2014

I’ve been doing static analysis for a while, first with C# and then with Python. I’ve even made an aborted attempt at a Python static code quality analyzer (pynocle, I won’t link to it because it’s dead). About a year ago we set up Sonar (http://www.sonarqube.org/) to analyze the Python code on EVE Online. I’m here to report it works really well and we’re quite happy with it. I’ll talk a bit about our setup in this post, and a future post will talk more about code metrics and how to use them.

Basic Info and Setup

Sonar consists of three parts:

  • The Sonar web interface, which is the primary way you interact with the metrics.
  • The database, which stores the metrics (Sonar includes a demonstration DB, production can run on any of the usual SQL DBs).
  • The Sonar Runner, which analyzes your code and sends data to the database. The runner also pulls configuration from the DB, so you can configure it locally and through the DB.

It was really simple to set up, even on Windows. The web interface has some annoyances which I’ll go over later, and sometimes the system has some unintuitive behavior, but everything works pretty well. There are also a bunch of plugins available, such as for new widgets for the interface or other code metrics checks. It has integrations with many other languages. We are using Sonar for both C++ and Python code right now. Not every Sonar metric is supported for Python or C++ (I think only Java has full support), but enough are supported to be very useful. There are also some worthless metrics in Python that are meaningful in Java, such as lines in a file.

The Sonar Runner

I’ll cover the Runner and then the Website. Every night, we have a job that runs the Runner over our codebase as a whole, and each sub-project. Sonar works in terms of “projects” so each code sub-project and the codebase as a whole have individual Sonar projects (there are some misc projects in there people manage themselves). This project setup gives higher-level people the higher-level trends, and gives teams information that is more actionable.

One important lesson we learned was, only configure a project on the runner side, or the web site. An example are exclusions: Sonar will only respect exclusions from the Runner, or the Web, so make sure you know where things are configured.

We also set up Sonar to collect our Cobertura XML coverage and xUnit XML test result files. Our Jenkins jobs spit these out, and the Runner needs to parse them. This caused a few problems. First, due to the way files and our projects were set up, we needed to do some annoying copying around so the Runner could find the XML files. Second, sometimes the files use relative or incomplete filenames, so parsing of the files could fail because the Python code they pointed to was not found. Third, the parsing errors were only visible if you ran the Runner with DEBUG and VERBOSE, so it took a while to track this problem down. It was a couple days of work to get coverage and test results hooked into Sonar, IIRC. Though it was among the most useful two metrics and essential to integrate, even if we already had them available elsewhere.

The Sonar Website

The Website is slick but sometimes limited. The limitations can make you want to abandon Sonar entirely :) Such as the ability to only few metrics for three time periods; you cannot choose a custom period (in fact you can see the enum value of the time period in the URL!). Or that the page templates cannot be configured differently for different projects (ie, the Homepage for the ‘Entire Codebase’ project must look the same as the Homepage for the ‘Tiny Utility Package’ project). Or that sometimes things just don’t make sense.

In the end, Sonar does have a good deal of configuration and features available (such as alerts for when a metric changes too much between runs). And it gets better each release.

The Sonar API

Sonar also has an API that exposes a good deal of metrics (though in traditional Sonar fashion, does not expose some things, like project names). We hook up our information radiators to display graphs for important trends, such as LoC and violations growth. This is a huge win; when we set a goal of deleting code or having no new violations, everyone can easily monitor progress.

Summary

If you are thinking about getting code metrics set up, I wholeheartedly recommend Sonar. It took a few weeks to get it to build up an expertise with it and configure everything how we wanted, and since then it’s been very little maintenance. The main struggle was learning how to use Sonar to have the impact I wanted. When I’ve written code analysis tools, they have been tailored for a purpose, such as methods/functions with the highest cyclomatic complexity. Sonar metrics end up giving you some cruft, and you need to separate the wheat from the chaff. Once you do, there’s no beating its power and expansive feature set.

My next post will go into more details about the positive effects Sonar and the use of code metrics had on our codebase.

3 Comments

Maya and PyCharm

15/01/2013

Recently at work I had the opportunity of spending a day to get Maya and PyCharm working nicely. It took some tweaking but we’re at a point now where we’re totally happy with our Maya/Python integration.

Remote Debugger: Just follow the instructions here: http://tech-artists.org/forum/showthread.php?3153-Setting-up-PyCharm-for-use-with-Maya-2013. We ended up making a Maya menu item that will add the PyCharm helpers to the syspath and start the pydev debugger client (basically, the only in-Maya instructions in that link).

Autocomplete: To get autocomplete working, we just copied the stubs from “C:\Program Files\Autodesk\Maya2013\devkit\other\pymel\extras\completion\py” and put them into all of the subfolders under: “C:\Users\rgalanakis\.PyCharm20\system\python_stubs” (obviously, change your paths to what you need). We don’t use the mayapy.exe because we do mostly pure-python development (more on that below), and we can’t add the stubs folder to the syspath (see below), so this works well enough, even if it’s a little hacky.

Remote Commands: We didn’t set up the ability to control Maya from PyCharm, we prefer not to develop that way. Setting that up (commandPort stuff, PMIP PyCharm plugin, etc.)  was not worth it for us.

Copy/Paste: To get copy/paste from PyCharm into Maya working, we followed the advice here:  http://forum.jetbrains.com/thread/PyCharm-671 Note, we made a minor change to the code in the last post, because we found screenshots weren’t working at all when Maya was running. We early-out the function if “oldMimeData.hasImage()”.

And that was it. We’re very, very happy with the result, and now we’re running PyCharm only (some of us were using WingIDE or Eclipse or other stuff to remote-debug Maya or work with autocomplete).

————————

A note about the way we work- we try to make as much code as possible work standalone (pure python), which means very little of our codebase is in Maya. Furthermore we do TDD, and we extend that to Maya, by having a base TestCase class that sends commands to Maya under the hood (so the tests run in Maya and report their results back to the pure-python host process). So we can do TDD in PyCharm while writing Maya code. It is confusing to describe but trivial to use (inherit from MayaTestCase, write code that executes in Maya, otherwise treat it exactly like regular python). One day I’ll write up how it works in more depth.

3 Comments