goless- Golang semantics in Python

21/04/2014

The goless library https://github.com/rgalanakis/goless provides Go programming language semantics built on top of Stackless Python or gevent.* Here is a Go example using channels and Select converted to goless:

c1 = goless.chan()
c2 = goless.chan()

def func1():
    time.sleep(1)
    c1.send('one')
goless.go(func1)

def func2():
    time.sleep(2)
    c2.send('two')
goless.go(func2)

for i in range(2):
    case, val = goless.select([goless.rcase(c1), goless.rcase(c2)])
    print(val)

While I am not usually a Go programmer, I am a big fan of its style and patterns. goless provides the familiarity and practicality of Python while better enabling the asynchronous/concurrent programming style of Go. Right now it includes:

  • Synchronous/unbuffered channels (send and recv block waiting for a receiver or sender).
  • Buffered channels (send blocks if buffer is full, recv blocks if buffer is empty).
  • Asynchronous channels (do not exist in Go. Send never blocks, recv blocks if buffer is empty).
  • The select function (like reflect.Select, since Python does not have anonymous blocks we could not replicate Go’s Select statement).
  • The go function (runs a function in a tasklet/greenlet).

goless is pretty well documented and tested, but please take a look or give it a try and tell us what you think here or on GitHub’s issues. I’m especially interested in adding more Go examples converted to use goless, or other Go features replicated to create better asynchronous programs.**


*. goless was written at the PyCon 2014 sprints by myself, Carlos Knippschild, and Simon Konig, with help from Kristjan Valur Jonsson and Andrew Francis (sorry the lack of accents here, I am on an unfamiliar computer). Carlos and myself were both laid off while at PyCon- if you have an interesting job opportunity for either of us, please send me an email: rob.galanakis@gmail.com

**. We are close to getting PyPy support working through its stackless.py implementation. There are some lingering issues in the tests (though the examples and other ‘happy path’ code works fine under PyPy). I’ll post again and bump the version when it’s working.

2 Comments

Global Glob

4/04/2014

I am cleaning out my drafts and found this two year old post titled “Globals Glob” with no content. The story is worth telling so here we go.

There was a class the EVE client used to control how missiles behaved. We needed to start using it in our tools for authoring missiles with their effects and audio. The class and module was definitely only designed (I used the term loosely) to run with a full client, and not inside of our tools, which are vanilla Python with a handful of modules available.

My solution was the GlobalsGlob base class, which was just a bag of every singleton or piece of data used by the client that was unavailable to our tools. So instead of:

serviceManager.GetService('someservice').DoThing(self.id)

it’d be:

self.globalsGlob.DoThing(self.id)

The ClientGlobalsGlob called the service, but FakeGlobalsGlob did nothing. The GlobalsGlob allowed us to use the missile code without having to rewrite it. A rewrite was out of the question, as it had just been rewritten, except using the same code. (sigh)

Unsurprisingly, GlobalsGlob was super-fragile. So we added a test to make sure the interface between the client and fake globs were the same, using the inspect module. This helped, but of course things kept breaking.

This all continued until the inevitable and total breakdown of the code. Trying to use the missile code in tools was abandoned (I think it was, I have no idea what state it’s in). This was okay though, as we weren’t using the missile tools much after those few months. GlobalsGlob served its purpose, but I will never be able to decide if it was a success or failure.

No Comments

Planet Mars, the simple feed aggregator

30/03/2014

In December, I spent some time polishing up a new fork of the old PlanetPlanet Python feed aggregator, named Planet Mars: https://github.com/rgalanakis/planet-mars . I chose the name to contrast it with Planet Venus, a successful PlanetPlanet fork. My goal for Planet Mars is to make it as simple as possible, to be the best aggregator for the very common use case. It’s based on my experience running a PlanetPlanet-based river at http://tech-artists.org/planet. That feed has been powered by Planet Mars for the last few months.

The main features are:

  • Jinja2 templating, in addition to the original htmltmpl engine. Old templates still work, new templates can use a better language.
  • Parallelization of feed updating, so things are many times faster.
  • The code is stripped down and much easier to understand and improve.
  • Planet Mars can be installed via pip, so instead of having to place your customizations in its package directory, you can pip install the GitHub repo and import it as a module into your own code (you can see the Planet TechArt repo as an example).
  • Planet Mars otherwise very similar to PlanetPlanet, making switching to PlanetMars very easy :)

Please take a look and tell me what you think, especially if you are an original Planet user.I’m very open to improvements and suggestions. If there’s any demand, I can put this on PyPI as well.

3 Comments

Why I love blogging

29/03/2014

I started to write a post about how I missed multithreading and speed going from C# to Python. I ended up realizing the service we built which inspired the post was poorly designed and took far more effort than it should have. The speed and multithreading of C# made it easier to come up with an inferior design.

The service needed to be super fast, but the legacy usage pattern was the problem. It needed to run as a service, but then we realize it’ll be running as a normal process. This is what happens when you focus too much on requirements and architecture instead of delivering value quickly. You create waste.

I wouldn’t have realized all of this if I didn’t sit down to write about it.

No Comments

Configuration in Host the Docs

28/03/2014

With Host the Docs, I chose to eschew configuration files and use a sphinx-style “conf.py” approach (I have previously written about how I like this approach). If a conf.py file is found, it is used for configuration, allowing it to override default configuration options. I also allow configuration through environment variables, where the environment variable name is the same as the conf.py variable but with HTD_ prepended.

So for example, in hostthedocs/getconfig.py, I have code like the following:

try:
    import conf
except ImportError:
    conf = None

def get(attr, default):
    result = os.getenv('HTD_' + attr.upper())
    if result is None:
        result = getattr(conf, attr, None)
    if result is None:
        result = default
    return result

welcome = get('welcome', 'Welcome to Host the Docs!')
# Other configuration values

I was initially going to use a json or yaml file, but always find their contracts a bit unclear, especially when paths are involved in the configuration (if the config uses relative paths, what are the paths relative to?). It also involves more code.

I was really happy with how using Python for configuration worked out in Host the Docs.

1 Comment

Introducing Host The Docs

22/03/2014

A month or so ago I created Host the Docs, and it’s been quietly running at work some success, so I figure it’s time to talk about it. The code is on GitHub at https://github.com/rgalanakis/hostthedocs, and here’s a description from the readme:

Host the Docs is a simple way to host static code documentation. Its main use is as a self-hosted server for your organization’s private documentation. Better alternatives are available for open source projects, such as Read the Docs or Github Pages.

Host the Docs was created after a long day of banging my head against the wall trying to get Read the Docs set up with private GitHub repositories, and having helped develop a plugin to get it to work with Perforce previously. What the world needed was a way to easily host documentation from several projects, from any source or language or SCM.

Seriously, let other people generate their own docs, I just want to Host the Docs!

It’s super easy to get set up, and requires no database. You can even just serve everything through Flask, if you don’t want to set up a webserver like Apache or nginx. It’s pretty well tested and configurable and thoroughly documented. We had it running on a new Linux VM in just a few minutes, and use it to host HTML documentation for Python and C# projects (we used Sphinx to generate the Python docs and Doxygen for C#).

I encourage you to try it out if you are interested in painless documentation hosting! I’m also very open to fixes/changes/pull requests, as long as it doesn’t overcomplicate things.

No Comments

Who is using PyPy?

11/03/2014

I brought up the idea of using PyPy for an internal service we are building at work, and was asked what actual projects are using PyPy? I had no answer and couldn’t find one. Are there are well known projects using PyPy? Do you know of anyone that is using it to run small servers, even? I find PyPy hugely exciting, and I know it’s current limitations and issues (as well as its benefits), but I’m wondering if anyone’s used it yet and what their experience has been.

7 Comments

Being amazed by software development

1/03/2014

I am continually amazed by the state of software development. I am amazed at how broken things seem to be, and I’m amazed at what powerful tools we have to fix things.

A few weeks ago, I struggled to find a documentation hosting solution. We have an internal version of Read the Docs which took far too much effort to get set up (dependency problems, creating a new SCM backend) and administrate, but Read the Docs doesn’t work with private Github Repos. After some hacking, we still couldn’t get it to work. So I looked at hasdocs.com, which promised to host our docs in the cloud, but was totally broken. I looked around at other solutions and was tearing my hair out. I was amazed that no good solutions already existed, and was more amazed that the best solutions are somewhere on the spectrum of broken (no offense here to RtD).

At the end of the day, it occurred to me my use case was incredibly simple: just provide an index page for (already generated) HTML documentation. Let people POST their HTML files and some metadata, and just dynamically generate the index based on what’s available. I was going to run this behind a firewall, so I didn’t need to worry about security. Hell I could even get away with no database, and just read the filesystem.

The next day (a Saturday), while my son and wife napped, Host the Docs was born. It took 1.5 hours to create an initial working version, and then I spent a few hours here and there polishing things up. It was painless to deploy, and I did some more improvements after that. Throughout this experience, a few things struck me:

  • I’m amazed by frameworks like Flask and Bootstrap. You can create a reasonable site in no time that is totally maintainable. At no point was I “hacking” to get something up and running, it was instead a very small first version I was able to iterate on.
  • I’m amazed by Linux. I have to use Windows at work, but feel like I develop faster on my Linux Mint netbook as I do on my Windows 7 workstation. The power at my fingertips is divine. It makes me mad at Windows.
  • I’m amazed how well some software works the same way I’m amazed at how broken some software is. Software is truly evolving; for every Flask, there are confusing and broken web frameworks.
  • I’m amazed how much time having autonomous teams can save. It would have taken a day or more to deploy HTD if I had to go through IT to provision a normal virtual machine (I doubt that frustration is unique in our industry). Instead, by giving teams an AWS budget and not centrally controlling things, HTD was live in minutes.

I’ll post more fully about Host the Docs later. I just wanted to express my satisfaction before it wore off :)


(BTW, I’m aware how similar this sentiment is to Jeff Knupp’s story of building Bull: Python and Flask are ridiculously powerful. Probably not coincidentally it is also a story of using Flask :)

6 Comments

Using Sonar for static analysis of Python code

15/02/2014

I’ve been doing static analysis for a while, first with C# and then with Python. I’ve even made an aborted attempt at a Python static code quality analyzer (pynocle, I won’t link to it because it’s dead). About a year ago we set up Sonar (http://www.sonarqube.org/) to analyze the Python code on EVE Online. I’m here to report it works really well and we’re quite happy with it. I’ll talk a bit about our setup in this post, and a future post will talk more about code metrics and how to use them.

Basic Info and Setup

Sonar consists of three parts:

  • The Sonar web interface, which is the primary way you interact with the metrics.
  • The database, which stores the metrics (Sonar includes a demonstration DB, production can run on any of the usual SQL DBs).
  • The Sonar Runner, which analyzes your code and sends data to the database. The runner also pulls configuration from the DB, so you can configure it locally and through the DB.

It was really simple to set up, even on Windows. The web interface has some annoyances which I’ll go over later, and sometimes the system has some unintuitive behavior, but everything works pretty well. There are also a bunch of plugins available, such as for new widgets for the interface or other code metrics checks. It has integrations with many other languages. We are using Sonar for both C++ and Python code right now. Not every Sonar metric is supported for Python or C++ (I think only Java has full support), but enough are supported to be very useful. There are also some worthless metrics in Python that are meaningful in Java, such as lines in a file.

The Sonar Runner

I’ll cover the Runner and then the Website. Every night, we have a job that runs the Runner over our codebase as a whole, and each sub-project. Sonar works in terms of “projects” so each code sub-project and the codebase as a whole have individual Sonar projects (there are some misc projects in there people manage themselves). This project setup gives higher-level people the higher-level trends, and gives teams information that is more actionable.

One important lesson we learned was, only configure a project on the runner side, or the web site. An example are exclusions: Sonar will only respect exclusions from the Runner, or the Web, so make sure you know where things are configured.

We also set up Sonar to collect our Cobertura XML coverage and xUnit XML test result files. Our Jenkins jobs spit these out, and the Runner needs to parse them. This caused a few problems. First, due to the way files and our projects were set up, we needed to do some annoying copying around so the Runner could find the XML files. Second, sometimes the files use relative or incomplete filenames, so parsing of the files could fail because the Python code they pointed to was not found. Third, the parsing errors were only visible if you ran the Runner with DEBUG and VERBOSE, so it took a while to track this problem down. It was a couple days of work to get coverage and test results hooked into Sonar, IIRC. Though it was among the most useful two metrics and essential to integrate, even if we already had them available elsewhere.

The Sonar Website

The Website is slick but sometimes limited. The limitations can make you want to abandon Sonar entirely :) Such as the ability to only few metrics for three time periods; you cannot choose a custom period (in fact you can see the enum value of the time period in the URL!). Or that the page templates cannot be configured differently for different projects (ie, the Homepage for the ‘Entire Codebase’ project must look the same as the Homepage for the ‘Tiny Utility Package’ project). Or that sometimes things just don’t make sense.

In the end, Sonar does have a good deal of configuration and features available (such as alerts for when a metric changes too much between runs). And it gets better each release.

The Sonar API

Sonar also has an API that exposes a good deal of metrics (though in traditional Sonar fashion, does not expose some things, like project names). We hook up our information radiators to display graphs for important trends, such as LoC and violations growth. This is a huge win; when we set a goal of deleting code or having no new violations, everyone can easily monitor progress.

Summary

If you are thinking about getting code metrics set up, I wholeheartedly recommend Sonar. It took a few weeks to get it to build up an expertise with it and configure everything how we wanted, and since then it’s been very little maintenance. The main struggle was learning how to use Sonar to have the impact I wanted. When I’ve written code analysis tools, they have been tailored for a purpose, such as methods/functions with the highest cyclomatic complexity. Sonar metrics end up giving you some cruft, and you need to separate the wheat from the chaff. Once you do, there’s no beating its power and expansive feature set.

My next post will go into more details about the positive effects Sonar and the use of code metrics had on our codebase.

3 Comments

What a powerful thing metaprogramming is!

5/02/2014

While editing a chapter of my book, I was introducing the concept of metaprogramming using Python’s type function. It occurred to me that I had already introduced metaprogramming several chapters earlier when introducing decorators.

Defining a function within another function is as important to my programming as bread is to French cuisine. I began thinking of all those cultures without wheat; all those languages without, or with newly added support for, metaprogramming. I have never done serious development in a language without anonymous functions, closures, and reflection.

It was exciting to think of a coming generation of programmers who are in my shoes (I started programming relatively late in life), who would be inherently comfortable with passing functions. It was exciting to realize where languages are going, keeping static typing but removing the explicit part. It was exciting to think of how flexible, expressive, and powerful languages have become.

It also allowed me to think of less flexible languages and what they’ve been able to achieve. I am lucky to be programming now, but surely each programmer before me felt the same about those before them. More will feel the same after and about me. Really my luck is to be part of what is still such a new and remarkable part of the human endeavor.

All of this feeling from type and decorators. What a powerful thing metaprogramming is!

2 Comments

Switch to our mobile site