My new pyjamas (python->javascript/html)

by Rob Galanakis on 24/06/2011

I mentioned I used pyjamas for building my content aggregator UI.  Now that the UI is built, and I’m happy with it, I feel more confident weighing in more strongly about pyjamas.

Pyjamas is awesome.  There, I said it.

I’m not going to go deep into what pyjamas is: There are FAQs and tutorials for that on their website.  I’ll concentrate on why I enjoyed using pyjamas over every other framework I looked at- including QT and wx, and I enjoyed it more than using WPF and WinForms with C#, too.

First, pyjamas is  written well.  It is based directly on Google Web Toolkit, and the generally well-written API works.  It isn’t entirely ‘pythonic’, but I still prefer to it to what I’ve used of other frameworks.  The event system is a little kludgey, but I haven’t had any problems with it, really.  I generally knew what things did based on their name and how they would be done.  It all worked as expected, with a clear API with a minimal amount of redundancy and confusion (consider how many properties in WinForms are tightly coupled and how frustrating they can be to use and configure because of that).

It is of a manageable size.  I didn’t feel overwhelmed by new concepts and classes.  It contains a manageable number of things and amount of code.  I felt that after a few days, I had a really good grasp for what I was doing and what was available in pyjamas.

It is well documented.  For two reasons: first, there are amazing examples.  It speaks volumes about the team and language that such examples with relatively little documentation and comments can be so expressive and clear.  Second, because it mirrors GWT so closely, you can basically use the GWT API documentation verbatim (and the demo materials and tutorials available).  Once I cracked into the GWT docs and realized how close they were, I never really felt at a loss for information.

It didn’t require a designer.  I’ve ranted previously about what I think visual UI designer tools are doing to our children.  I never once felt the need to use a designer with pyjamas.  All the subclassing and composition that served me well in WinForms was better and easier in pyjamas.  All the layout just happened naturally and straightforwardly.  It just made me happy.

It uses CSS.  This is beautiful, really.  The truth is, I don’t think I’ve ever seen one person really use the styling options available in any web framework.  Styling is always done at the code level, even with XAML/QML- that is at the code level for me because there are so many fucking options and specifics, you need tool support or you’ll get something wrong (or forget lots of stuff).  CSS is dead simple, well documented, and tool support is ubiquitous- PyCharm even has it built in.  It was an absolute pleasure to perform the styling of my UI with CSS.

My entire UI, which is moderately complex, is less than 600 lines of Python.  Some of that is because I can use lambdas like a champ ;), but mostly that’s because 1) python is compact, 2) no designer, and 3) pyjamas is simple and expressive, and 4) all styling and configuration is done in CSS, which is even more compact and straightforward.  I’m beginning to cringe thinking about doing this type of thing in C#.

I wonder how my zealotry for moving to a JS/HTML application base would go over, and how it would work in context?  Hmmm, that seems perfect for a future post!

2 Comments

I booted up VS2010 today…

by Rob Galanakis on 22/06/2011

And I’m still waiting for it to load.  Even though I love all of Visual Studio’s features, my god, I forgot how big and heavy it is.  Having a light IDE and a system where I don’t need to create a new project (several more minutes) to just scribble some code like python has is nice.

Though I have faith that IDEs like VS will evolve somewhat more rapidly going forward, we’ll see if they get larger and more monolithic, or more feature-rich but componentized.

No Comments

Blog roll: Joe Duffy

by Rob Galanakis on 21/06/2011

Another awesome blog totally worth reading is by Joe Duffy, who is a Lead Architect at Microsoft, and an expert in concurrency, performance, and memory.  Calling it a blog is unfair to bloggers- he really doesn’t update it often.  But the articles he has on there are incredibly lucid and interesting, revealing information I’ve never read on any blog, anywhere (stuff you can generally only find in books and papers, except delivered much less formally and much easier to understand).

Definitely go and read these posts:

Why volatile is evil

Dense and pointer free (regarding memory performance)

The premature optimization is evil myth

Thoughts on immutability and concurrency

I could go on and on (and in fact, those are just his 4 most recent posts- they’re all gems), but almost every post is incredibly deep and worth reading.  So do what I do with all those inactive MSDN blogs: bookmark it, read through it in a weekend (or for Joe’s, a few weekends).

No Comments

Server/Client apps as an abstraction exercise

by Rob Galanakis on 20/06/2011

My last couple personal/work projects have involved creating remote services and local clients (as well as interfacing with other remote services).  It’s been an interesting exercise in creating well-abstracted interfaces, because 1) network transfer is slow, so you want to limit the amount of data you send, and 2) serializing objects for transfer across processes is much more limiting than calling methods in process (especially the lack of ability to pass lambdas/methods/callbacks easily).  So there’s a lot of pressure to develop well abstracted public APIs for a service.  Here’s what I’ve found:

  1. The client shouldn’t have to replicate the server.  IE, the client could ask for every piece of data the server has and run its own queries.  But that would be terribly inefficient.  The server, instead, needs to offer a balance between getting raw content and having methods for common queries (like everything with a certain tag or category).
  2. So you end up with a ‘dirtier’ service API than if you were designing a single class- the APIs are inherently larger and have more ‘helpers’ than an in-process class, where the consuming code could just query the data itself to get what it needs (all other things being equal).  There are ways to combat this- like the exposed API being on multiple classes- but the easiest way is just to have the API on one class (the implementations, of course, can and should be broken up to adhere to the Single Responsibility Principle as much as possible).
  3. Asynchrony is difficult between server/client.  At least initially, consider making a synchronous service API and hide the asynchrony behind a background thread.  This isn’t ideal because your threads are blocking.  But it is much simpler.  Once things are stable and working, consider making an asynchronous service API.
  4. Pick the right framework/setup.  Know your needs.  Is this an internal system?  Is it a local-only service?  Must it communicate across languages or just within one language?  Service/client frameworks are very complex and the simpler you can make your needs, the better.

In the end, the ideal is to have an API that provides everything the client needs but only what the client needs.  “Everything should be made as simple as possible, but not simpler,” says Einstein.  It would also be useful to read up on RESTful APIs (the Wikipedia article is good and probably covers a lot of what you’d learn by trial and error), as well as understanding how things like XMLRPC, JSONRPC, sockets, and CGI work, even (and maybe only) in just an introductory sense.

No Comments

Code metrics, requiring a culture of quality

by Rob Galanakis on 18/06/2011

Last time I went over how adhering to things like code quality metrics that are objective and ‘scientific’ is the key to creating and sustaining a strong codebase.  The difficulty comes with actually implementing that process and behavior wherever you work.  There is no shortage of obstacles:

1.  Convoluted process.  The unfortunate truth is that many of us work at a place with a convoluted submit/build/deploy process.  This is either so brittle that it is difficult to augment, or so complex it has a large dedicated build team.  Either is a problem because the hurdles to making process changes like setting up code analysis as a part of the submit/review/build process are very high.

2. Shame.  If more senior or lead developers are not willing to do this, it is unlikely it will be done.  This is compounded by the fact that the implicit blame of what the analysis reveals is on the shoulders of the more senior devs, so they may be less willing to do it.

3. Disagreement.  Fundamentally, there is a breed of developer that is opposed to Augmented Coding.  They tend to endorse (and exhibit) high proficiency with simple tools (text editors, commandline), and actively oppose more sophisticated GUI programs or tools.  It will be very difficult to get this type of programmer to change his or her view.

4. Scheduling.  No discussion of any change would be complete without talking about scheduling issues.  Someone has to do this work, train people, etc.  So like all things, this needs to be scheduled, or ninja’d in if possible (impossible if you have a bad process).

These problems combine in pretty frustrating ways.  And unfortunately I really don’t have a solution.  There’s no glorious ending to this blog post that will tell you how to overcome these problems.  Even if I’ve overcome all of these problems personally, these are cultural problems, and cultural problems are notoriously specific.  Ultimately I think it comes down to hoping that you can get key people- leads, build engineers- on board with the necessity of having code quality metrics as part of your pipeline.  That’s the most important thing you can do to make sure that this time, I’m going to do it right.

No Comments

Code metrics, the only ‘right constant’

by Rob Galanakis on 17/06/2011

I wrote recently about the experience of running a code analysis tool on a codebase and hinted at the difficulties involved with refactoring the problems.  There are far smarter people than me who have given much more thought to the technical problems and strategies involved.  I want to explore, instead, the cultural and human problems involved.

I doubt there’s a developer who wrote the first line of code in a codebase without thinking, ‘this time I’m going to do it right.’  And I also doubt there are many developers who are working in a codebase who aren’t thinking, “If I get a chance to start from scratch, I’m going to do it right.”  So how is it possible that these two sentiments exist simultaneously?

The answer is another paradox- early development is done without enough rigor and is done with too strict adherence to early established principles.  Ie, the rigor that is used is applied towards principles that fail in the long run.  Over several years, languages change, technologies become available or obsolete, developers grow and evolve, etc.- and the codebase becomes larger.

The way to ‘do it right,’ then, is to establish what is right as a constant and what is correct right now.  In all of software development, the only thing that I can think of that is ‘right as a constant’ is code quality metrics- things that are not subjective (like code reviews), and backed up with empirical evidence about effects.  If code quality metrics are not part of your process, your codebase is likely to fail.  As a codebase grows, so does the liklihood that future development is under the paradigms already existing in the codebase.  The problem is, these paradigms have no certainty that they will yield good code.  In fact, chance is they will be directly at odds with more widely established and accepted principles and paradigms that have evolved or appeared after the codebase started.  This is the nature of the myopia and bubble that forms at any sizeable development house.

The only way to fight this is to apply the steady force of the ‘right as a constant’ factors to a codebase.  If you can do this, you’ll always be at a more agile place, so you can refactor more easily.  Anecdotal evidence would indicate that any other strategy is futile.

Have I missed any other possible ‘right as a constant’ things that can be implemented?

Next up: What implications does this have for culture?

1 Comment

Blog roll: CodeBetter.com

by Rob Galanakis on 16/06/2011

I am going to start making some blog posts about other blogs when I don’t have time for bigger posts.  The first blog up is www.codebetter.com, which covers a variety of code quality and .NET topics.  It is contributed to by a number of people, so there’s a pretty good flow of excellent topics and posts.  It has quickly become one of my favorite blogs to read, and though it focuses on .NET, the lessons are applicable to any language.

Here are some recent highlights:

LINQ Intersect 2.7 times faster with HashSet

db4o’s no primary keys

On partitioning .NET code

Back to basics: Usage of static members

No Comments

Relearning python, part 8: Over the hump

by Rob Galanakis on 15/06/2011

I did it.  As I was finishing yesterday’s blog post, I finally got my project working, and exposed on the internet.  Now I that things are finally figured out, I can document and test it.

I ended up writing a process that runs a socket/ZeroMQ based service, which is long-running and persistent.  I have my web UI, written with pyjamas, that uses jsonrpc through CGI.  The CGI service/handler (which runs on the server, obviously) opens a brief connection to the persistent service to run whatever method call it was asked to call, and return the result.  Until I deploy it on the actual tech-artists.org server, I have my router port-forward incoming connections to my machine.  So I’ve used my service from my Droid successfully ;)  I have no idea if this is a terrible design, but it serves my needs well enough.

I’ve been really impressed with pyjamas.  I think I’ve gotten over the learning hurdle, and am starting to compose together a pretty nice UI.

Hopefully I can finish this project in the next couple weeks, and move on to other things as I just tighten things up and improve it.

Once I got over the hump, I went back to enjoying things again.  I could write code with confidence, and feel like I was learning and making progress, rather than just trying things arbitrarily.

That should wrap up the real work for this ‘relearning python’ series- I’m not sure that I’ll reach any more epiphanies, and I’m now pretty comfortable with the switch from C# to python.  I’ll make sure to wrap things up with a conclusion post or two, as promised.

No Comments

Relearning python, part 7: Which way is up?

by Rob Galanakis on 14/06/2011

Weeks after beginning python, I have hit a forest on a plateau- the speed of learning and discovery has slowed, and I’m getting confused and discovered.  I’m beginning to think I won’t finish my current project- I am sure deploying it on a server is going to be another exercise in frustration and hope I can even make it to that point before my life gets busy.  Here’s what I’m struggling with.

1. Changing my mind.  I was very much in the habit, in the static-typing world, to declare simple immutable data types.  This seems unnecessary and uncommon in python- instead, use dictionaries.  This makes sense- the onus of immutability is on the caller in python (in general), and the dynamic nature means having those simple data types doesn’t buy anything over a dictionary (especially if you use a bunch-like object, you will probably want to write your own though).  The problem is that a) this makes refactoring more difficult- this sort of linking is where static typing shines, so changing- and then removing- these data types is never ‘foolproof’, just quick and sometimes with minor breakages that unit testing hopefully finds.

2. Unittesting.  I am writing tests and actually enjoy it.  And I am getting better.  Mostly my problem is with running tests.  I am struggling along the spectrum of everything manually (set up to run the .py file with the tests run if __main__) and completely done through the IDE w/ nose (pycharm -> Python nosetests).  I actually found out a lot of my tests weren’t being run, somehow.  I’ve now switched it to be the manual setup, so I can have better control and learn more about what’s going on.  That’s a problem with convention-driven systems, I guess, is that the implications can get too confusing for noobs like me.

3. Web programming.  I didn’t approach this project correctly.  I bit off way more than I can chew.  New technologies are fine with new concepts, usually, but I’m using new technologies that use ‘old’ technologies, so the beginner-oriented tutorial material I need is not there.  So it’s been incredibly frustrating.  I now have my ZeroMQ-based service running, with a JSONRPC CGI service to field requests from my pyjamas-created web UI.  I feel I’ve acquired a dangerously unfocused lack of information about way too many things (socket programming, CGI, RPC, javascript, etc.), and the system is held together with bubblegum.  Maybe not, but I feel that way.

I am struggling more from not having people to ask questions to than I am because of anything to do with python or the web.  Part of the problem is I’m so unfamiliar with certain things that I don’t even know what questions to ask.

I look forward to starting at CCP and having people to field more general python questions, and I also feel myself getting over the hump and getting a hang of the web stuff enough to actually make something.

1 Comment

Relearning python, day 6: Please serve me!

by Rob Galanakis on 10/06/2011

I wrote last time about all the difficulty I had with deciding on a UI framework.  Well, it turns out that was nothing compared to trying to get a server up and running.

I suppose I should say first that I have no experience doing web programming.  None.  I have some experience writing server/client apps, that were all WCF based.  I expected difficulties with python, but nothing like this.

My goal was clear- develop an RPC-like service interface that I can call from pyjamas.  So, first thing, I implement an XMLRPC server, based on SimpleXMLRPCServer.  Wow, that was easy!

Wait.  pyjamas cannot use XMLRPC because it isn’t a supported module.  They suggest JSONRPC.  Alright, well, no biggie.  Right?

Well unfortunately, there’s nothing so simple as SimpleXMLRPCServer for jsonrpc.  Even SimpleJSONRPC, because nothing I tried on the client seemed to work.  So began evaluating a dozen modules, looking for what should have been very simple.

I looked at the pyjamas JSONRPC example.  But it used CGI, and I need a persistent service.  I am not a Unix programmer and do not know how to, and don’t care to, set up mod_python, which apparently allows you to run python in Apache or some shit that I don’t want to care about.

So I looked at anything with jsonrpc in the name.  There are well over 20 modules that hit ‘jsonrpc’ at pypi.python.org.  Including ones that seem to conflict (how many are used with ‘import jsonrpc’?  At least two, from what I saw).  Keep trying, nothing works.

Alright, let’s up the ante.  Let me try stackless- can you even make a server in that?  Well I couldn’t, and I can’t remember the problems.  Well let’s look at Twisted.  Wait, zope.interfaces needs to be installed.  OK, got that.  Alright, following directions, and… fuck, that example didn’t work.  Why?  Can’t remember.  Alright, let me look at PyZeroMQ.  Easy install doesn’t work on Windows.  Shocker.  I’ll use the MSI.  There we go.  Whoops, that example didn’t work.  Wait, I need to download and build ZeroMQ myself?  Are you fucking kidding me?  Here we go.  Wow, build problems.  So surprised.  Huh?  Wait, this examples working.  Did I have to build/install?  I don’t know.  Sort of working now… ok, lots of good examples that seem to be mostly working.  I can do this.  I am scared to see what’ll happen when I move from a local to a remote server, though.  Well, let’s clean up some of this crap… wait, I am scared to think about the dependency tree here, and have no idea what is or is not safe to remove.

Ultimately I have no idea if ZeroMQ is what I want or will do just what I need, but it was the only thing that ended up working reasonably well (I only struggled with it for 1-2 hours before using it successfully!).  My guess is I will have several more hours trying to figure out how to make it work with pyjamas.  But at least this whirlwind of problems, so rapid and severe I can’t even recall all the issues, has subsided for now.

This situation is unforgivable.  It is representative of a huge ‘lie’ about python in particular and a problem with the OSS situation in particular.  I’ll have more to say about it soon.

No Comments