Why CCP is still using Python 2

by Rob Galanakis on 9/01/2014

We at CCP are maybe the heaviest users of Python in videogames, though I have no data to back that up. (I’ll also use this opportunity to say this is a personal blog post, I am in no official capacity here) We use it in the client and server of both EVE Online (PC) and DUST 514 (PS3), and nearly all of our internal infrastructure uses it. What’s stopping us from upgrading at least some portions to Python 3?

Everything. Even if Python 2 weren’t good enough (it is), even if a hundred compelling features were added to Python 3 (there aren’t), even if Stackless was available for Python 3 (it isn’t), we still probably wouldn’t switch. Because literally everything conspires against enterprise employees who want to upgrade any significant codebase. Let’s go over some of those things.

We have our own localization solution inside EVE and the unicode/str bugs have been worked out. Oh, the solution is a nightmare, and our string handling is often a mess, but that just means changing it would be even more difficult. So there’s no real external product need, and internal products and tools usually aren’t localized. But wouldn’t it be great to change it and get rid of that technical debt and simplify things?

Sure it would be great to get rid of that tech debt! As it would for literally every area of our 11+ year old product. There is a limited amount of technical debt we can clean up, and none of it has to do with string handling or any Python 3 features. We just removed a custom importer we’ve wanted to remove for years, which paves the way for other technical debt cleanup. But we’re at least a year from another codebase-wide cleanup, of which there are many to do (let’s remove our dozen remaining builtins, please!). When low-level or non-value-adding work involves convincing people all the way up to the corporate/business level, there are very few people who can organize that sort of thing. I generally prefer that energy is spent on activities that general more value to our engineers.

We have relatively few automated tests. We’ve made some great progress on testing in the past couple years, especially for new or refactored/rewritten code, but there’s absolutely no way to uncover and fix Python 3 upgrade bugs easily. We have “extensive” manual regression tests, but it takes time to get builds to those testers and I imagine the regression test cycle would take months to get everything worked out. It would be a hard sell to QA and the turnaround time on bugs would mean the upgrade process would take months, not weeks.

The only place a Python 3 upgrade gets traction is with the core of the Python community here. And unfortunately we spend most of our capital by trying to improve our own systems, training our Python programmers, and even keeping other developers using Python.*

We can get backports of libraries developed for or in the stdlib of Python 3. If they weren’t provided in a sustainable way by others, we’d just copy them in and keep them locally.

We keep a critical eye on performance. While Python 3 seems fast enough for us now, that’s a pretty new development, and we absolutely will not regress on performance. So that could mean upgrading our codebase and still not be able to use Python 3, since we won’t fully know until we do it. Lovely!

Middleware, the scourge of OSS! We use Autodesk Maya, which uses 2.6 (soon 2.7) for our art pipeline. We would need to write a large chunk of our code to support Python 2 and 3 (an ongoing inconvenience), but also need infrastructure to test them in both (added work but not a huge deal, since we already do this in some cases for 2.6/2.7). But middleware is an easy excuse to stop an upgrade. Even upgrading middleware like Maya, which carries little risk, can take over a year.

Ultimately we’ve been too sloppy and monolithic** (like a large number of enterprise users, I’d imagine). Our products make money. Our developers are supposed to be working on those products and thus making money. We didn’t do a good job balancing technical and business needs, and besides our codebase is really old (EVE Online’s codebase is still an ancestor of the original codebase that EVE released with in 2001).

All of this means the “default” Python for me is Python 2. It is my “go to” language when I’m working on something where I have the option. I haven’t used Python 3 for any real work, so there’s still a threshold to cross. Familiarity and comfort. I don’t even have Python 3 installed at work. And most of the people I know are working in Python 2, so there’s a synergy of inertia. I have no problem with Python 3*** and plan to roll it out at work somewhere (it is a 2014 goal of mine), though I am intrigued with all the Python 2.8 (both vanilla and Stackless) discussions going on recently.


* There are some defections to other languages, and causing inconvenience to these customers on the fence is only more likely to push them into other languages. I have nothing against using other languages inhouse (in fact I wholly support it), I just want to make sure they’re used for the right reason. There’s value in being able to work in other people’s code without having to learn a new language and environment, and there’s value to doing as much as possible in high level languages.

** This has in many ways changed, with lots better automated testing and code quality overall, but legacy code is still the majority. We’re just producing less new legacy code.

*** I also understand that upgrading applications like EVE Online was not part of the Python 3 adoption strategy.

rob.galanakis@gmail.com

There are 27 comments in this article:

  1. 9/01/2014Abulurd says:

    Why don’t you just write it in C?

    Also: keep up the great work, I’m rooting for all developers!

  2. 9/01/2014Richard Sharp says:

    This is the first article that mentions what I would regard as the crunch argument. What would an upgrade cost and how do I sell it. The software I maintain is getting on for 15 years. My problem middleware is Twisted. I’ve done a lot of work on testing recently. And the unicode stuff has also been dealt, but it’s still a mess.
    The argument I would be faced with is “instead of porting to Python 3 we could rewrite it in c#”.

    A good article.
    Greetings,
    Richard Sharp

  3. 9/01/2014leonardo says:

    Given the dynamic typing model of Python of course every bigger code change is primed for a major disaster. I wouldn’t want to change a running Python (or any other dynamic language) code base either. Well, actually, I wouldn’t want to greenfield anything in a dynamic language to be honest.

    CCP is in a evolutionary dead end with their careless love for Python. Let’s hope they somehow can avert the impending disaster because I actually like playing EvE.

    Also: What the hell were you thinking making Dust 514 PlayStation only? It’s like you don’t want it to successful.

  4. 9/01/2014mjfgates says:

    Python is a complete failure! You should have picked one of the other, better languages available back in 2001… like Visual Basic 6.0. Or Java, which back then, that would have been some kind of Java Beans thing with, um, Struts? Was Struts a thing yet?

    Yeah. Just think about maintaining an EVE written in VB 6, today. Wouldn’t that be great?…

  5. 9/01/2014king6cong says:

    I’m afraid that there will be no Python 2.8

  6. 9/01/2014mtelesha says:

    mjfgates: “Python is a complete failur!” Use Visual Basic LOL

    If Python is a failure why is it working now for this company and why is it a top 3 language? Keep your FUD out of the argument. This is an issue about longevity of code base and how to handle technology debt.

    The fact is Python 2.7 does everything they want and Python 3 doesn’t give them a return in value because they are fine with 2.7.

    Anyway someone will pick up Python 2.7 and maintain it. I am thinking RedHat. (This happened to Ruby on Rails LTS)

  7. 9/01/2014dpanda says:

    Feel free to correct me, I’m piecing together information from blog-read memory. From what I’ve gleaned about CCP’s stack over the years I assume that you’re running the server on stackless backed by MSSQL. Because of MSSQL, I also assume that you’re running the server on Windows.

    If my assumptions are correct, your usage of the language is very far from what could be considered standard usage, at least on the server side. As much as I like Python, it doesn’t sound like a rational choice, Python3 notwithstanding.

  8. 9/01/2014Amorano says:

    Rob, pretty much every reason you list is why we are in it @nickelodeon still as well (Autodesk being a large reason).

    Frankly, who cares about 3? Is it so much better that when all the re-writes and regressions testing is done, I can seriously say we just saved 100k? 1mil? over time?

    It just doesn’t make sense to fix what aint broke in some situations.

    What I would offer is simply make the NEW parts you can make, in something else (even PY3), instead of locking into the older IF YOU CAN.

    If not, again, unless it is shaving crazy man hours a week, what is the point?

  9. 9/01/2014mbien says:

    languages like python do not scale. You can be very productive in green field projects but don’t try to maintain the for a long period of time – once they end being green the problems will start. Just ask a java dev to refactor something, he will love it. It is like playing with legos and using IDE magic while hitting those automated tests from time to time. Ask a python or C dev to refactor something non trivial and he will hate you forever and will try to convince you that it is a very bad idea to do this at this stage of the project.

    I am feeling really bad for CCP having to stick with this language on server-side. Maintenance is becoming harder and harder, tools are not comparable to what others have (e.g the prev mentioned java camp) and being forced to write single threaded code on the server and even client side is just horrible if you take a look what hardware vendors sell. Eve is the only game i know where the client is literally freezing if you do certain things (e.g spamming lock target while not being in range) for a fraction of a second. Such things would never happen if concurrency would be used right. I wish you guys the best since i love Eve but i am constantly reminded of the shortcomings of the implementation tech while playing it.

  10. 9/01/2014Holms says:

    I’ll be short, if you don’t have unit test – don’t even dream about refactoring. If, at least, you’d have non-monolithic application, than it would be easier, but for now I don’t even imagine how you’d go with py3. In your case using c/cpp would be a really wise solution. To use python on a long term there’s requirements like tests and soa, or else – nothing but pain

  11. 9/01/2014Sanders says:

    This may be of interest:

    http://nuitka.net/posts/re-about-python-3.html

    By the way nuitka is awesome! http://nuitka.net/pages/overview.html

  12. 9/01/2014Rob Galanakis says:

    Thanks for the comments everyone!

    leonardo and mbien: I fundamentally disagree and it will be the subject of a subsequent post.

    dpanda: Python was decided because when CCP was started, it is what the people here knew. Since then, we’ve been happy with it. I can’t say the same of Windows. Yes it’s definitely a bit of a unique stack. I suspect we are the largest corporate Python-on-Windows users.

  13. 9/01/2014Rems says:

    Have you thought about using pypy?

  14. 9/01/2014Dzmitry says:

    Google has a good habit of rewriting from scratch its projects/subprojects every 3-10 years. It helps not only to clean up codebase, it also becomes a good security review and raises employees morale (everyone wants to write something new, or using new technologies). Also, it helps to keep relatively small variance in technologies used, preventing “language zoo”.

    Although I must admit Google has enough resources for that. But it’s very tricky business to estimate where you’d spend more resources — on rewriting from scratch or maintaining 2001ish antiquities for some more years.

  15. 9/01/2014Jóhann Haukur says:

    CCP made a great choice going with the powerful combination of C++ and Python back in the days.
    With full control over the stack and a pretty powerful exposure model, there’s not much that’s needed in Python 3 that can’t be created in house.

    While the environment that surrounds a technical stack is fresh and alive, the stack can remain relevant. So while the python community at large continues to be 2.x driven, developing in Python 2.7+ will feel fine for developers.

    Great article!

  16. 9/01/2014Trevor says:

    I believe Sims 4 might be giving you a run for your money on being the heaviest user of python in video games. Perhaps we should compare line counts or something! :D

  17. 9/01/2014marv says:

    “we’ve been happy with it. I can’t say the same of Windows.”

    Sooo… you are finally tired of windows? I assume that your little love story with MSSQL (and probably the whole lot of ASP apps and maybe internal tools) are still making you stick with it? Or is there any tiny tiny chance we see EVE native on Linux in some years? ;-)

  18. 10/01/2014K900 says:

    Looks like there is actually a Stackless version of Python 3. http://www.stackless.com/wiki/Download
    Not sure if it’s stable or not, because their “release” process is weird, but it’s there.

  19. 10/01/2014bryancole says:

    I think everyone understands that Enterprise Users are not going to want to upgrade to py3 in a hurry. These are not the people the python community needs to worry about. For enterprise users for whom python2 works fine, python2 is not going away and they should just stick with this. The key thing is, highly bespoke enterprise codebases are not used widely by other thirdparty users (outside the enterprise in question). Hence, if said enterprise sticks with py2 this isn’t hindering anyone else from upgrading. It’s the pypi library/package authors who are the key people who need to upgrade, to allow *users* the option if there are key features in py3 they want (and there are, for me at least).

  20. 10/01/2014Abulurd says:

    Can anyone chime in with a suggestion for the best possible language to write EVE Online in, if you knew what you were going for in the long run? What language would best be suited to write a solid gaming universe?

  21. 10/01/2014Jóhann Haukur says:

    Abulurd I’d aim to have my key gaming services communicate in a language agnostic manner. So the game universe could be split into separate projects that could be developed and released on different tracks. Language could then be taken out of the equation, allowing the best tool for the job.
    There are some key technologies that support this kind of design, such as ZMQ.

  22. 10/01/2014Stephan says:

    I don’t understand why some people are (apparently) measuring the success of Python 3 against what version long-standing Python users are using for existing code bases. It’s also not surprising to me that some third party libraries are lagging in converting to Python 3 because the people that might do that conversion are tied to Python 2 applications that base on that library. The history of computing is absolutely littered with examples of this — it just doesn’t make sense to change platforms very often or at all for a legacy application. But the fact that lots of code is still Fortran or Cobol hasn’t stopped other languages from appearing and succeeding any more than this effect is going to cause problems for Python 3 in the long run. While I look forward to some day writing new code with Python 3, and I do think it’s a strong improvement, I can hardly complain being “stuck” on Python 2 for now. In 30 years of programming I’ve never worked with a better platform.

  23. 10/01/2014airtonix says:

    @mbien

    I would like to know (since I feel a bit inadequate on the topic you’re describing), what makes it un-scalable (and what do you mean by un-scalable) ?

  24. 10/01/2014DrTune says:

    Python is a terrific language – speaking as a relatively recent convert (3-4 years ago) having used most of the others. Lack of unit testing in your codebase is obviously an annoyance for you guys but it’s nothing to do with Python itself. I can only assume the “why don’t you use C/CPP” people are joking or very inexperienced; it’s massively less productive; more time-consuming to understand, maintain and debug.

    Personally I’m very, very happy with Python 2.7; have used it on several projects (including one that’s currently running great with 34 million users on a dozen or so EC2 boxes). It’s absolutely my preferred language – not least because everyone’s code is naturally so readable. Like you I can’t see a need to migrate to Python 3.

  25. 10/01/2014Kristján says:

    Ah, Rob, We’ve had Stackless Python 3 for years now. IIRC we started with 3.1 in SVN. We’ve been a bit lazy providing regular installs of late, but we’ve been maintaining Stackless Python 3.2 and 3.3 in Hg and 3.4 will be there when CPython 3.4 is released.
    Check out http://hg.python.org/stackless

  26. 21/01/2014Seth Williams says:

    Hi Rob,

    It is very good to see you guys using Python in your videogames. Keep up the good work and I would like to wish all the developers with good luck. We too at http://www.fireboxtraining.com/ are all about sharing the wonders of Python with anyone interested.

  27. 3/08/2014Legacy Code is Cancer says:

    […] and “Poor programmers, being forced to program and all”, respectively. Granted, there were some valid arguments, and those tend to show up in projects that are so absurdly large, but the Py2 vs Py3 issue […]

Write a comment:


six + 3 =