Armin Ronacher

I just released Version 0.5.1 of Werkzeug, the swiss army knife of Python web developers. Tons of bugfixes went into this release making it what 0.5 should have been: rock solid.

Grab it while it's hot from the python package index, read the changelog, or the documentation.

Last week I was in Birmingham (UK) for this year's EuroPython. It's the first time I was at EuroPython and also the first time there was an offline sprint on Pocoo projects. I went there with Georg from the Pocoo Team and Alexander who mainly works on PyPy and MoinMoin.

The first two days we mainly spent sprinting on Zine which will soon have another release with tons of changes. Unfortunately the network connectivity at the Conservatoire where the tutorials, sprints and talks took place was very bad and we also had some troubles finding tables and chairs. Also, the first two days there were quite a lot of students there, practicing their instruments. And after a few minutes you get sick of people playing the scale up and down.

Despite that, we got a lot of changesets into Zine and the next release will come with notification support and a blogger importer because of that.

Birmingham itself is quite boring, I won't talk about the typical English food here, I suppose everybody knows how it tastes :)

Language archeology

One of the most interesting talks was “Language Archeology†by Bruce Eckel. While he's know to be “the Java†or “the C++†guy, he seems to like Python a lot and gave a good overview of the development history of various programming languages and why they work like they do. He painted a pretty scary picture of C++, a language I just recently learned to love. On the other hand he described the Python world as if it was utopia which made me believe he did not use Python long enough. Maybe I also misunderstood him, but I did not remember any criticism at all.

Also what surprised me is that he mentioned garbage collection as a good feature in Python and Java. Many C++ programmers tend to dislike forced garbage collection because they consider explicit resource management using RAII significantly superior.

Also I did not really understand what he meant with “C++ supporting both exceptions and traditional, C-error handlingâ€. I know that many people dislike C++ error handling because it integrates badly into existing code, but that's just how things are. Even in Python you have “traditional†error handling by checking return values of functions and retrieving error descriptions from somewhere else. If you're looking hard enough you will find libraries that feel a lot like C, even in Python. I suppose his talk was targeted towards Python programmers with no C++ background at all, but it was quite confusing what exactly he meant with it.

The Science of Computing

Tony Hoare (the developer of the Quicksort algorithm) held a talk about the differences between science and engineering. From what I recall he declared science being that idealist thing, that should be valid forever and engineering being a short term solution for commercial projects. Scientists are trying to achieve perfection by creating proof-able algorithms and be certain about the result. A software engineer tries to wage the data to create an acceptable, stable result. At the end of the talk he was talking about how to break a software project into smaller parts to be able to analyze them and proof the algorithms. With the help of Moors law and improving tools computer science will once be the reliable form of all engineering professions because what's created does not decay by the forces of nature.

One of the questions from the audience was how to work with the marketing department to get specifications that are implementable. Tony Hoare's answer was that this is the engineers job because neither does the marketing department understand programming nor what the costumer wants.

Lightning Talks and Discussions

What I like the most about any conference are lightning talks and discussions with other attendees. Here a brief overview about some of the most interesting lightning talks or notes from discussions I had with some of the folks there:

Hatta — a distributed Wiki

Hatta by Radomir Dopieralski (aka “the sheepâ€) is a wiki, written in Python with the help of Werkzeug and mercurial that is easy to set up and distributed with the help of the underlaying DVCS (hg). The idea and implementation is pretty cool and I can see a bright future for the project. I suppose we will see more of those in the near future as they are the next logical step with the rise of decentralized tools. (Just waiting for the ideal decentralized bug tracker)

Psyco 2.0 — a JIT from the ashes of PyPy

Christian Tismer revived the Psyco project, the oldest JIT for Python. It was unmaintained for a long time and did not support more recent features of Python such as generators (not so recent, but they were indeed unsupported). He showed a benchmark of the builtin python property class and a pure-python reimplementation to demonstrate how much psyco can speed up property access. On average, there was a 100x improvement for that particular case. There should be a release in -2 days, so it can't take that long.

Nice to see that there is work done on Psyco again because it suddenly stopped when Armin Rigo started working on PyPy which was intended to replace CPython + Psyco. But unlike PyPy, there is some real world benefit in Psyco so any progress on that is welcomed. Also it means that there will be some interesting speed races between Unladen Swallow, CPython + Psyco and IronPython in the near future.

py.test — putting the fun back in testing

Another codespeak (codespeak is the name of the home of PyPy, the py-lib, lxml and other libraries) project that was unmaintained for too long is py.test. Once the great alternative to the builtin Python unittest everybody loved, now the forgotten magical support library in the pylib. However Holger Krekel now took over the maintenance and it could become interesting again. Not only will there be a Python 3 port but also improved support for distributed and multi-python-version testing and probably a less-magic release as a standalone library and without the wird import behavior.

Seems like nose got a contestant again.

SourceForge on Python and back in Action

It was the first time at EuroPython I met Mark Ramm in Person. He is currently the BDFL (seems like they have to work on the “for life†part) of TurboGears and working at SourceForge a website I nearly had forgotten. But during EuroPython they rolled out their latest iteration of the website and I think they are on the right track now. The company behind it recently acquired ohloh, a community open source project I always found interesting. From what Mark told me the focus of SourceForge will probably change (to the better) soon and we can expect the website to become more relevant again.

And probably the coolest thing is that parts of the website are now powered by Python and also Jinja2.

The Importance of Names

Less cool but not less interesting is another thing I learned at EuroPython. People tend to love or hate projects not only based by code quality and documentation, but also by the name. I've heard from multiple people that they did not even had a closer look at Werkzeug, solely based on the name. While I will not rename the library because of that, I learned from that to chose my names more wisely in the future.

Guido on the Moon

My personal highlight of whole EuroPython was the non-working Skype link to Guido van Rossum. Because Guido did not appear there in person (again) they tried to get his video on the projector and his voice out of the loudspeakers. However the network connection was in such a bad state that there was a lag of multiple seconds between him and us. After someone joked if Guido was on Google's secret moon base and no answer came, the guys on the stage decided to go over to questions and answers. However after they finally delivered the audience's question about unittest in the standard library, Guido answered that he wouldn't be able to breath there which made the whole audience laugh hard. Apparently there were not only a few seconds in delay but nearly a minute.

When they finally switched over to asking the questions using the Skype chat and guido started answering, the notebook showed a “battery status low†notification and before someone was able to plug the power chord in, the thing went into power saving mode. (And before anyone asks: yes it was a linux machine)

Pocoo Sprints

Unfortunately we did not have that many people working on Pocoo code, even though there were quite a few people from #pocoo which however also worked on other projects. Ali and Ronny sprinted on Pida (which loves you). But I was very happy to see Michael Foord sprinting on RST support in Zine. His changes will soon land in Zine tip. Also again many thanks to all the patches we've accumulated over the last weeks and over the internet during the sprints. I was able to pull some of them already, many more will follow soon.

Also: I finally got my contributor agreement to CPython code submitted to the PSF. Raymond Hettinger should be happy now :)

Georg found a way today to get the names for a variable in Python. Motivated by this incredible hack I gave porting runkit_return_value_used from PHP to Python a try. What this function does in PHP is finding out if the return value of a function is used or not.

So if you're evil you could write a function like this in PHP:

<?php

function the_today() {
  $today = date("l jS \\of F Y h:i:s A");
  if (runkit_return_value_used())
    return $today;
  echo $today
}

?>
<p>Today is: <?php the_today(); ?></p>

This would print the current date if the function is invoked without assigning the return value to something but will return the current date as string otherwise. I have no idea how this works in PHP but you can have that functionality in Python as well.

import sys, dis

def rvused():
    frame = sys._getframe(2)
    remaining = frame.f_code.co_code[frame.f_lasti:]
    try:
        next_code = remaining[ord(remaining[0]) >= dis.HAVE_ARGUMENT
                              and 3 or 1]
    except IndexError:
        return True
    return ord(next_code) != dis.opmap['POP_TOP']

You can then use it like this:

>>> def foo():
...  if rvused():
...   print "my return value is used"
...   return 42
...  print "my return value is not used"
... 
>>> def test():
...  print foo()
...  foo()
... 
>>> test()
my return value is used
42
my return value is not used

The implementation of the function is pretty simple actually. The rvused function goes two stack frames back, which is the stackframe of the function calling rvused. Then we look at the next bytecode after the last executed instruction (f_lasti) which should be the one that handles the return value of our function. If the bytecode is POP_TOP it means we have an unused value on the stack the virtual machine has to throw away. In that case we let the function return False to signal an unused return value.

How this works becomes obvious if we look at the bytecode for test:

>>> dis.dis(test)
  2           0 LOAD_GLOBAL              0 (foo)
              3 CALL_FUNCTION            0
              6 PRINT_ITEM          
              7 PRINT_NEWLINE       

  3           8 LOAD_GLOBAL              0 (foo)
             11 CALL_FUNCTION            0
             14 POP_TOP             
             15 LOAD_CONST               0 (None)
             18 RETURN_VALUE        

The last instruction in both cases is CALL_FUNCTION. The next one is PRINT_ITEM in the first case which means that python will print the last value on the stack, which is the return value of the function. In the second case the opcode after the function call is POP_TOP which tells the VM to throw away the result.

Oh. And please don't use that in your code. It was just a stupid experiment to find out if it was possible to port that abomination of a function from PHP to Python. Maybe it helps for debugging though.

I'm probably not telling news if I tell you that Wave is awesome. And that for so many reasons. Quite frankely, I was blown away by the demonstration not only because the application felt like a desktop thingy, what really convinced me that this is something new was the fact that Wave is not only an application on Google servers but also an open protocol. It's nice that the application is open source as well, but dammit: It's an open protocol. Imagine the possibilities.

Concept wise the idea is nothing new. We've had that realtime text based communication with very old ICQ clients as well (at least reddit say that). However people change and after a decade things look different again. Wave would integrate into my personal workflow like if it was developed exactly for what I do. I love E-Mail but when it comes to figuring out problems with others nothing beats IRC for me. Just for comparison: the #pocoo IRC channel has ~100 people online at the same time but the activity on the mailinglist approximates zero.. The main problem I have with IRC is however that it's hard for other people to benefit from earlier discussions. We do have a logging bot in the channel and the logs are publicity available but they are painfully to use as reference for your own problems. So IRC has problems but the nearly real-time behaviour makes it easy to help me fix other people's code or discuss implementations etc.

Wave extends both mail and instant messaging. A wave can be composed like a mail and people can comment to it like they would do on IRC. Imagine how effective mailinglists would work that way. You have a problem and describe it in a new wave. Then someone that has some ideas can add a new comment to it. If you happen to be online at the same time you can write to it like it was an IRC session. And that would even scale to a large number of concurrent users because the users are not writing to a central log like they do on IRC but to a topic. In this case the problem you just outlined in that new wave.

So far the only Google products I use and like are the search, maps and youtube. I don't care about gmail because I like to have things in my own hands. I hate centralized systems as much as the next guy. With Android things at Google somehow changed. They started their first large-scale open source project. I always thought that would stop there, but they were doing that again for Chrome as well. And with Wave the whole concept of Open Source at Google went into a new direction. Wave is not only a product but also a protocol you can implement yourself. And if people accept it there should be some alternative implementations available soon. And not only is the specification open, also the implementation Google uses. That makes it incredible easy to get started and setup your own Waves.

I have no idea why Google is doing that, but I think it's great. They are not forcing the users to be present on their servers. They even said in the introduction video explicitly that if you're sending a Wave from one user to another user on the same Wave instance, the messages would never leave the original server. There are very few companies that would dare to give users that much freedom.

You probably have to be a big company to be able to give away your product like this, but I seriously hope other companies will follow Google in that regard. I know Microsoft is slowly getting the concept of open standards and I can only hope they will try to become a bit like Google here. (even though I'm pretty sure this won't happen)

EuroDjangoCon was awesome! I came back yesterday evening after 5 days of great talks and sprints from Prague. For the week there I had to postpone some exams for university, but it was totally worth it.

If you haven't been there, some of the best moments of the week in recap:

Talks

Zed Shaw, known for his extravagant presentations decided to use a terminal as his presentation tool that wrote the “slides†to stdout and where then projected to the wall. At the same time he was broadcasting the talk to twitter, sentence by sentence. However seems like he breached the Twitter API limits half way through so the talk is now available in full length in his blog: EuroDjangoConf2009 Keynote All Over Your Twitters. The main point he made during that talk was that it's important for programmers to have interests besides programming that emphasize on the concept of creativity and practice.

The talk by Simon Willison was probably the one that motivated me the most. I have to admit that I often when the easy way replacing a whole part of the Django infrastructure for projects with external libraries instead of trying to find a solution for the problem and improving Django. Simon's talk was addressing one of the problems I had with Django, if not the problem I had with Django. The infamous DJANGO_SETTINGS_MODULE. And I'm not talking about that being an environment variable but that magic singleton named django.conf.settings that is loaded once and nearly impossible to change at runtime.

His talk was addressing mainly replacing middlewares and the URLconf with view-like callables that take the request and return a response, but we where brainstorming with Adrian in an open space later and soon we've had a healthy discussion going on about breaking Django internally into multiple systems that can be optionally configured separately and changed at runtime in a thread-safe manner. The slides from his talk are online on slideshare: Django Heresies.

Joe Stump from digg was talking about large data sets and rethinking the stack. For me this was very interesting because the only time I was personally working with largish data was when I was working for Plurk and it wasn't that big back then :) I personally still have the feeling that the larger the data the weirder the code, but maybe that's how things are. I suppose as a result of his and Michael Malone's talk people will start working on message queues for Django. I was unable to find the slides for the data set talk online, but the one about Rethinking the Stack is online as keynote source: Rethinking_the_Stack.key.zip. Michael's talk was about scaling Django at Pownce and was interesting as well. You can find the slides online on his website: Scaling Django.

Adrian was briefly showing some of the history of Django before it was open sourced in a lightning talk. Even though I was using Django since the first open source release it was interesting as hell what happened before that. Did you know there were like 8000 changesets before the first release? That means that until recently there was more stuff happening before the open source release than afterwards :-) Also I didn't know that Ian Bicking was responsible for Django no longer using code generation (yay!). Really seems like he has his hands in every Python project out there.

There was a lot more, but I think these were my favorites.

Sprints

Sprinting on Django was great fun. The first day we were 50 people in the offices of centrum holdings, the company that provided the room for the sprints. Honza was ordering 30 pizzas and Jacob came with two or more bags of chips and sweets. I got 9 tickets closed so far, 6 still in the pipeline. Alex and I also refactored the file system in Django slightly so that it doesn't require monkey patching of __class__ and replacing all methods in subclasses, like it was the case before.

There was also a dedicated pinax sprint, but I didn't take part of that.

Prague

Prague rocks. What do I have to say more? When I stepped of the train I draw 2000 crowns from my account (~75€) and I was unable to spend all of it the week even though we went drinking and eating every single day. The party was in a place where they had beer taps in the middle of the table and the number of liters the table drew was shown in a scoreboard on the wall. Unfortunately we often went in really large groups which made it hard to get to the really interesting places (like a Jazz bar), but now we know better.

What else?

I didn't take any pictures myself (and I have the ability to avoid photographers) but there are some nice galleries on flick. Next time they should decide on a better tag than edc which is hard to search for on flickr.

Can't really wait for djangocon in Portland.