Archive of articles classified as' "Programming"

Back home

goless now on PyPI


goless is now available on the Python Package Index: . You can do pip install goless and get Go-like primitives to use in Python, that runs atop gevent, PyPy, or Stackless Python. You can write code like:

channel = goless.chan()

def goroutine():
    while True:
        value = channel.recv()
        channel.send(value ** 2)

for i in xrange(2, 5):
    squared = channel.recv()
    print('%s squared is %s' % (i, squared))

# Output:
# 2 squared is 4
# 3 squared is 9
# 4 squared is 16

I’ve also ported the goless benchmarks to Go, for some basic comparisons to using goless on various Python runtimes (PyPy, CPython) and backends (gevent, stackless):

Thanks to Rui Carmo we have more extensive examples of how to use goless (but if you’ve done or read any Go, it should be relatively straightforward). Check them out in the examples folder:

And just a compatibility note (which is covered in the docs, and explained in the exception you get if you try to use goless without stackless or gevent available): goless works out of the box with PyPy (using in its stdlib) and Stackless Python. It works seamlessly with gevent and CPython 2.7. It works with PyPy and gevent if you use the tip of gevent and PyPy 2.2+. It will support Python 3 as soon as gevent does.

Thanks and if you use goless, I’m eager to hear your feedback!

No Comments

goless Benchmarks


I benchmarked how goless performs under different backends (goless is a library that provides a Go-like concurrency model for Python, on top of stackless, PyPy, or gevent). Here are the results, also available on the goless readthedocs page:

Platform Backend   Benchmark      Time
======== ========= ============== =======
PyPy     stackless chan_async     0.08400
CPython  stackless chan_async     0.18000
PyPy     gevent    chan_async     0.46800
CPython  gevent    chan_async     1.32000
~~~~~~~~ ~~~~~~~~~ ~~~~~~~~~~~~~~ ~~~~~~~
PyPy     stackless chan_buff      0.08000
CPython  stackless chan_buff      0.18000
PyPy     gevent    chan_buff      1.02000
CPython  gevent    chan_buff      1.26000
~~~~~~~~ ~~~~~~~~~ ~~~~~~~~~~~~~~ ~~~~~~~
PyPy     stackless chan_sync      0.04400
CPython  stackless chan_sync      0.18000
PyPy     gevent    chan_sync      0.44800
CPython  gevent    chan_sync      1.26000
~~~~~~~~ ~~~~~~~~~ ~~~~~~~~~~~~~~ ~~~~~~~
PyPy     stackless select         0.06000
CPython  stackless select         0.38000
PyPy     gevent    select         0.60400
CPython  gevent    select         1.94000
~~~~~~~~ ~~~~~~~~~ ~~~~~~~~~~~~~~ ~~~~~~~
PyPy     stackless select_default 0.00800
PyPy     gevent    select_default 0.01200
CPython  stackless select_default 0.19000
CPython  gevent    select_default 0.25000

The trends are that PyPy with its built-in stackless support is fastest, then Stackless Python (2-5x), then PyPy with gevent (5-10x), and finally CPython with gevent (15-30x).

Remember that these are benchmarks of goless itself; goless performance characteristics may be different in real applications. For example, if you have lots of C extensions, CPython/stackless may pull ahead due to stack switching.

If you’re using goless for anything, I’d love to get some benchmarks, and find out how it’s working, so please comment below, or create a GitHub issue.

Disclaimers: It’s possible that goless is inefficiently using gevent, but the backend-specific code is so simple I doubt it (see goless/goless/ These benchmarks are all done against Python 2.7. Also I am no benchmark master (especially with PyPy) so there may be problems, look over goless/ for benchmark code. These were done mostly for curiosity.

No Comments

Deploying a C# app through pip


“If we bundle up this C# application inside of a Python package archive we can deploy it through our internal CheeseShop server with very little work.”

That sentence got me a lot of WTF’s and resulted in one of the worst abuses of any system I’ve ever committed.

We had a C# service that would run locally and provide an HTTP REST API for huge amounts of data in our database that was difficult to work with.* However, we had no good way to deploy a C# application, but we needed to get this out to users so people could start building tools against it.
I refused to deploy it by running it out of source control. That is how we used to do stuff and it caused endless problems. I also refused bottlenecks like IT deploying new versions through Software Center.

So in an afternoon, I wrote up a Python package for building the C# app, packaging it into a source distribution, and uploading to our Cheese Shop. The package also had functions for starting the packaged C# executable from the installed distribution. The package then became a requirement like anything else and was added to a requirements.txt file that was installed via pip.

What I initially thought was an unsightly hack ended up being perfect for our needs. In fact it caused us to eliminate a number of excessive future plans we had around distribution. I certainly wouldn’t recommend this “technique” for anything more than early internal distribution, but that’s what we needed and that’s what it did so effectively.

Getting something up and running early was extremely useful. It’s important to walk the line between the “we need to do several weeks of work before you see any result” and “I can hack something together we’re going to regret later,” especially for infrastructure and platform work. If code is well-written, tested, documented, and explained, you shouldn’t have qualms about it going into (internal) production. If the hackiness is hidden from clients, it can be easily removed when the time comes.

* Reworking the data was not an option. Creating this service would have allowed us to eventually rework the data, by providing an abstraction, though.


An Unfoolish Consistency: Introducing PEP8 to a legacy codebase


Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is most important.

The EVE source code, being initially developed before PEP8 existed, was based on Microsoft’s C++ style. Mostly this is manifested in UpperCamelCase function and method names, mixedCase parameter and variable names, and file headers and class/method delimiters. There was also a style guide that was probably larger than PEP8 (though included a bit more than basic style).

Today, most new code is PEP8 compliant. “What!?” I hear you say. “A foolish consistency is the hobgoblin of little minds!” I agree, and I’ll get back to that at the end. But first, what was involved in introducing PEP8?

The move started when I asked my team if anyone wanted to keep using the headers and delimiters. No one did. So we took out the headers and delimiters before we did any modifications to a file.

A while later, I asked all the programmers on the project if anyone felt strongly about keeping the existing style. No one did. So we started writing new packages with PEP8. The rationale was that our code was already calling code using lowercase_underscore in the stdlib, so it’d be no different for that same code to call a relatively independent, though still project-related, library.

Yet a while later, I started slipping some PEP8 code into modules or packages that weren’t PEP8. I was worried people would get upset. No one did.

And finally, in preparing some basic utility libraries for usage on other projects, I converted the function and method names to PEP8. This involved find-and-replace or CamelCase aliases. I bet you can guess who cared about these “superfluous” edits, or objected to the fact that parameter and variable names remained mixedCase? No one did.

Emerson’s quote “a foolish consistency is the hobgoblin of little minds” is often used as a reason to not change existing code to PEP8, or introduce it into a non-PEP8 codebase. But the quote means the opposite. Foolish consistency” is being unable to change your mind because it would contradict something you’ve believed or said. So PEP8 has it so right but so wrong: the foolish consistency is to keep things the way they are, even though we know what the right thing to do is.*

* I would recommend introducing PEP8 incrementally. I also suggest keeping things backwards compatible when it is more prudent, such as if the changes are expansive, or you do not control all the code. I also don’t suggest being a stylistic nitpicker when it comes to non-functional aspects such as line length. There may also be times to break the rules (a class that behaves like a function, or function that behaves like a class def). As always, good sense should be applied when considering any viewpoints on this blog.

No Comments

goless- Golang semantics in Python


The goless library provides Go programming language semantics built on top of Stackless Python or gevent.* Here is a Go example using channels and Select converted to goless:

c1 = goless.chan()
c2 = goless.chan()

def func1():

def func2():

for i in range(2):
    case, val =[goless.rcase(c1), goless.rcase(c2)])

While I am not usually a Go programmer, I am a big fan of its style and patterns. goless provides the familiarity and practicality of Python while better enabling the asynchronous/concurrent programming style of Go. Right now it includes:

  • Synchronous/unbuffered channels (send and recv block waiting for a receiver or sender).
  • Buffered channels (send blocks if buffer is full, recv blocks if buffer is empty).
  • Asynchronous channels (do not exist in Go. Send never blocks, recv blocks if buffer is empty).
  • The select function (like reflect.Select, since Python does not have anonymous blocks we could not replicate Go’s Select statement).
  • The go function (runs a function in a tasklet/greenlet).

goless is pretty well documented and tested, but please take a look or give it a try and tell us what you think here or on GitHub’s issues. I’m especially interested in adding more Go examples converted to use goless, or other Go features replicated to create better asynchronous programs.**

*. goless was written at the PyCon 2014 sprints by myself, Carlos Knippschild, and Simon Konig, with help from Kristjan Valur Jonsson and Andrew Francis (sorry the lack of accents here, I am on an unfamiliar computer). Carlos and myself were both laid off while at PyCon- if you have an interesting job opportunity for either of us, please send me an email:

**. We are close to getting PyPy support working through its implementation. There are some lingering issues in the tests (though the examples and other ‘happy path’ code works fine under PyPy). I’ll post again and bump the version when it’s working.


The manager’s responsibility to review code


I believe any technical leader has a responsibility to review all the code that goes into a codebase.* I am certainly not the only person to feel this way (Joe Duffy as MSFT and Aras Pranckevičius as Unity have said the same).

Furthermore, I don’t believe the responsibility to review code ends at a certain level. Everyone from an Engineering Manager to the CTO should be reviewing code as well. In my experience, I’m able to do thorough reviews for 5 to 8 people, and more cursory reviews for another 15 to 20.

Code review above a team lead level** is not about micro-management. A manager should never, ever be saying “we do not use lambdas here, use a named function instead.” Instead, try “do you think this would be more readable with a named function instead of a lambda?” Or maybe you say nothing, and observe what happens, and inspect what other code looks like. If lambdas are common in the codebase, either your opinions need more information, or you have done a poor job educating.

Code reviews by managers should be about getting enough information to manage and lead effectively.*** It keeps you up to speed about what is going on, not just in terms of tasks, but in terms of culture. Are people writing spaghetti? Are bad choices challenged? Are hacks put in? Is code documented? Are standard libraries being used? Are the other technical leads reviewing and leading their teams effectively? You can learn an incredible amount through code review, and you need this information to do your job of leadership and management effectively.

*: I believe all programming managers and leaders must be able to program. I find it shameful this needs to be said.

**: It should go without saying, but team leads should be reviewing every checkin on that team.

**: Code reviews are the genchi genbutsu, or the go and see part of Lean management.


Global Glob


I am cleaning out my drafts and found this two year old post titled “Globals Glob” with no content. The story is worth telling so here we go.

There was a class the EVE client used to control how missiles behaved. We needed to start using it in our tools for authoring missiles with their effects and audio. The class and module was definitely only designed (I used the term loosely) to run with a full client, and not inside of our tools, which are vanilla Python with a handful of modules available.

My solution was the GlobalsGlob base class, which was just a bag of every singleton or piece of data used by the client that was unavailable to our tools. So instead of:


it’d be:


The ClientGlobalsGlob called the service, but FakeGlobalsGlob did nothing. The GlobalsGlob allowed us to use the missile code without having to rewrite it. A rewrite was out of the question, as it had just been rewritten, except using the same code. (sigh)

Unsurprisingly, GlobalsGlob was super-fragile. So we added a test to make sure the interface between the client and fake globs were the same, using the inspect module. This helped, but of course things kept breaking.

This all continued until the inevitable and total breakdown of the code. Trying to use the missile code in tools was abandoned (I think it was, I have no idea what state it’s in). This was okay though, as we weren’t using the missile tools much after those few months. GlobalsGlob served its purpose, but I will never be able to decide if it was a success or failure.


Mike Bland’s profound analysis of the Apple SSL bug


Mike Bland has done a series of articles and presentations about the Apple SSL bug over on his blog. To be frank, it’s pretty much the only good coverage of the bug I’ve seen. He’s submitted an article to the ACM; crossing my fingers it gets printed, because it’s an excellent analysis. I’ve been frustrated it hasn’t gotten more traction.

Mike’s take is that this bug could have easily been caught with simple unit testing. It’s sad this has been so overlooked. We gobble up the explanations that do not force us to confront our bad habits. We can point and laugh at the goto fail (“hahaha, never use goto!”), or just shrug at the merge fail (“we’ve all mistakes merging”). It’s much more difficult to wrestle with the fact that this error- like most- is because we are lazy about duplicating code and writing tests.

I have no problem categorically saying anyone who doesn’t blame the SSL bug on lack of automated testing and engineering rigor is making excuses. Not for Apple but for his or herself.

No Comments

Planet Mars, the simple feed aggregator


In December, I spent some time polishing up a new fork of the old PlanetPlanet Python feed aggregator, named Planet Mars: . I chose the name to contrast it with Planet Venus, a successful PlanetPlanet fork. My goal for Planet Mars is to make it as simple as possible, to be the best aggregator for the very common use case. It’s based on my experience running a PlanetPlanet-based river at That feed has been powered by Planet Mars for the last few months.

The main features are:

  • Jinja2 templating, in addition to the original htmltmpl engine. Old templates still work, new templates can use a better language.
  • Parallelization of feed updating, so things are many times faster.
  • The code is stripped down and much easier to understand and improve.
  • Planet Mars can be installed via pip, so instead of having to place your customizations in its package directory, you can pip install the GitHub repo and import it as a module into your own code (you can see the Planet TechArt repo as an example).
  • Planet Mars otherwise very similar to PlanetPlanet, making switching to PlanetMars very easy :)

Please take a look and tell me what you think, especially if you are an original Planet user.I’m very open to improvements and suggestions. If there’s any demand, I can put this on PyPI as well.


Why I love blogging


I started to write a post about how I missed multithreading and speed going from C# to Python. I ended up realizing the service we built which inspired the post was poorly designed and took far more effort than it should have. The speed and multithreading of C# made it easier to come up with an inferior design.

The service needed to be super fast, but the legacy usage pattern was the problem. It needed to run as a service, but then we realize it’ll be running as a normal process. This is what happens when you focus too much on requirements and architecture instead of delivering value quickly. You create waste.

I wouldn’t have realized all of this if I didn’t sit down to write about it.

No Comments