That is until today, when I discovered the ‘qrencode’ utility. Qrencode does exactly what it says on the tin. It takes some text, encodes it as a QR Code and writes the result as a PNG file. It’s a simple command line tool, which opens up awesome scripting possibilities.

My use case is simply to send the URL of a web page that I’m viewing on my desktop to my phone, via QR Code. I found that qrencode was quite happy to write its output to stdout, which meant I could display the result directly using ImageMagick’s ‘display’ command, e.g:

$ qrencode http://blog.webworxshop.com -o - | display

You’ll notice that by default the QR Code is quite small. We can fix this by increasing the block size:

$ qrencode http://blog.webworxshop.com -s 10 -o - | display

Next, I wanted to automatically pull the text to encode from somewhere to save typing – the clipboard was an ideal candidate. Enter ‘xclip’. Xclip is a command line utility to read and write from/to the X system’s built in clipboard. I used bash’s backtick command substitution to grab text from the clipboard and encode it:

$ qrencode `xclip -o` -s 10 -o - | display

And there you have it, a simple one line command to generate and display a QR Code from the contents of the clipboard. I created a bash script containing the command and assigned it to a keyboard shortcut in Gnome (Ctrl-Shift-Q), so that sharing URLs to my phone is as simple as selecting the text in the location bar and hitting Ctrl-C followed by Ctrl-Shift-Q.

If you want to give this a try, you’ll need to install the utilities discussed, in Fedora these can be installed with:

$ sudo yum install qrencode xclip ImageMagick

Enjoy!

• Installation and Setup
• Site integration
• User interface
• Extensibility (API availability)
• Overall impressions (documentation, community, etc.)

The philosophical argument

As well as the obvious benefit (from a Freedom perspective) of using one less proprietary web service, there is also another reason that I switched away from Google Analytics. Basically, this was privacy. For a while I’ve been using technologies to limit the amount of data which leaks from my browser as I navigate the web, in order to reduce the amount of profiling of my web activities. This isn’t because I have anything to hide. I just don’t like the idea of large companies building up a huge database on me, without my permission. The upshot of this is that I found myself in the slightly hypocritical situation of blocking GA in my own browser, but using it to track others on my site.

The solution was obvious, remove GA from my site. However, I didn’t want to lose the valuable information that it provides me with. Also, I don’t have a problem with site owners collecting data that can help them, just with them sharing it with 3rd parties such as Google, who then build it into their larger profiling efforts. A quick search turned up Piwik which aims to provide a full featured GA replacement that you can run on your own server. Because site owners run their own instances, they remain in charge of their tracking of users, retain ownership of the data and best of all don’t give any data to Google.

With the aim of responsible and unobtrusive tracking in mind I’ve added a page to my site to allow users to Opt-out of the Piwik tracking by means of a cookie. The link is also accessible from the sidebar under the copyright notice. I’m afraid some of the text on that page is pretty difficult to see with my current theme, but I’m working on this. For now just uncheck the check box to opt-out.

Right, on to the main event, the actual review…

1. Installation and Setup

There’s actually not much to say here, which is because installation was ridiculously easy! I just downloaded the zip to my sever (with wget) and unzipped it into my server root directory. This produced a directory called ‘piwik’ and a ‘How to Install Piwik.html’ file, which if you point your browser at it will redirect you to the installation instructions. The rest of the installation was fairly simple, following the instructions I pointed my browser at the ‘/piwik/’ directory of my site and was greeted by the installer. Following this was really easy, you’ll need to create a MySQL database when prompted for the database info, but that’s about as hard as it gets. Towards the end you’ll be prompted to setup your site with Piwik which involves entering a few details about the site, then you’ll be provided with a snippet of JavaScript to add to your site template. Which leads me neatly into the next section…

2. Site Integration

I didn’t copy and paste the JavaScript into my template, instead opting to install the WP-Piwik addon for WordPress. This made the set up easy and also gave me a widget on my WordPress admin dashboard which gives me a nice overview of my site visits. As I already said I was also able to add a widget to the site to enable visitors to opt-out of tracking. This was also simple, just involving a copy and paste of a couple of lines of HTML from one of the settings pages into a WordPress page. Easy!

You can also integrate Piwik widgets with your site, by following the instructions in the documentation, this is a neat feature, especially if you have a custom start page set in your web browser (something which I have yet to get around to making).

I also investigated the campaigns functionality in order to track entries to my site from the RSS feed. This is really simple to use, all you have to do is append the query string ‘?piwik_campaign=NAME’, where NAME is the name of your campaign to the end of a URL, to have it show up under that campaign. I found that I could integrate this with WordPress pretty well by adding the following snippet of code to the functions.php file of my theme:

1
2
3
4
5
6
7

function piwik_track_feed($url)
{
     $url .= "?piwik_campaign=RSS";
     return $url;
}
 
add_filter("the_permalink_rss", "piwik_track_feed");

If you now check the URLs in your RSS feeds, they will all have the query string added and clicks will be attributed to the ‘RSS’ campaign in Piwik.

3. User Interface

The Piwik user interface is really nice. I’ve included some screenshots below, so that you can make up your own mind. It’s pretty similar to the GA user interface, only cleaner and all the AJAX stuff makes it feel really responsive. I also love the real time tracking widget, which is something GA totally lacks. The only bad thing about the UI is the requirement of Flash for the graphs. I hate Flash and it doesn’t have a reliable 64-bit Linux version, which means I only have it installed on my netbook. Oh, and before you ask, I tried it with Gnash and it didn’t work!

4. Extensibility

By extensibility, I was primarily interested in API access. There’s certainly no shortage of this with two APIs listed on the documentation page. One API is for performing tracking, which I didn’t need given my usage of the WordPress plugin. I looked instead at the analytics API, which allows you to access all the data through simple HTTP requests. I was able to write a simple Python script to email me my main statistics once a day, in about an hour (including working out how the Python email and smtplib modules work!). Performing an Piwik API call in Python is as simple as:

1
2
3

def getVisits(idSite, period, date):
    url = "%s/index.php?module=API&method=VisitsSummary.get&idSite=%d&perio$
    return json.load(urllib2.urlopen(url))

Of course, as it’s Python its ridiculously simple!

Of course, if you find something that you can’t do with the API (which is unlikely, because it seems to cover everything), the you can access the data in the database – because it’s in YOUR database. You can also back-up and secure your data exactly how you want to. This is something that GA just can’t compete with!

5. Overall Impressions

My impressions of Piwik as a project have been really good. The documentation is excellent and there seems to be a good community behind it. As a product its a pleasure to use, really easy to install and just works. The reliance on flash for the graphs is a bit disappointing, but perhaps this will change in the future as HTML5 matures. Here are the obligatory scores:

• Installation and Setup – 5/5
• Site Integration – 4/5
• User Interface – 3/5
• Extensibility – 5/5

Overall Score: 4/5

Verdict: If your currently using Google Analytics, stop it! (and use this instead)

• Keynote: “Creating Pythonic APIs” by Anthony Baxter, from Google. This was really well put together, informative and above all hilarious.
• “How Python is influencing Neuroscience Research” by Daniel Myall. Again good to see python being used for hardcore sciencey stuff!
• “Don’t Block the GUI” by Glenn Ramsey. An introduction to basic threading and futures, I knew most of this stuff, but it was still interesting.
• “Python in the Datacentre” by Matt Provost, from Weta Digital. Interesting to see the internals of how Python loads, it probably explains why Python is slow on embedded systems too.
• “Wikkid Design” by Tim Penhey. About the design of a distributed Wiki system for Launchpad. Some good take home messages around choosing modules (personal recommendation and good documentation rule!).
• Open Spaces (Track 1): “Why aren’t we using Python 3″, “NoSQL Databases” and “Robotics and UAVs”.
• “Demystifying Unicode” by Leon Matthews. I knew nothing about unicode before this, now I know how to use it and that I should be using it!
• “Teaching Computer Science with Python” by Carl Cerecke. An interesting look into the Canterbury CS Department’s moves towards Python for teaching.

That’s it for now, there will be more in my full round up, which I’ll write when I have time (sometime this week). Bye for now!

So here’s the talks I attended:

• Keynote (obviously) – “How I learned to stop worrying and love deployment” by Jacob Kaplan-Moss (a core Django developer). This turned me onto a few modules, which I need to take a look at some point.
• “Freeing the Cloud one (small) service at a time” by Francois Marier – about free web services and his service, Libravatar, in particular (Francois was also kind enough to help me out with a problem I was having with it right there and then, thanks Francois).
• “Building a distributed Key-Value store with Cassandra” by Aaron Morton from Weta Digital. This was interesting, though I think a lot of it went over my head!
• “Packaging and Virtual Environments” by Brett Wilkins (and another presenter, who’s name I can’t remember – so sorry to him!). This was really good and turned me onto distribute, which is a setuptools replacement. I love the image on their website:

• Lightning talks – Of which the one on Bottle was particularly interesting – I’ve been looking for a lightweight web framework to play around with, so I’ll give it a try.
• “5 Good Reasons for Automated Testing” by Roman Joost. Yeah, I really should be using automated testing!
• “An Opinionated Guide to what makes a Good Unit Test” by Michael Hudson Doyle. Reinforced the opinion expressed above!
• “Amazon Web Service: An Introduction” by Simone Brunozzi. This was quite businessy, but this stuff is definitely on my “must check out” list.
• “Python In Astronomy” by Ian Bond. I missed the beginning of this but it’s awesome to see Python being used to do some hardcore science – finding planets no less!

Well that’s my quick summary of day one, hopefully there’ll be more good stuff tomorrow, so I’ll ‘see’ you then.

Side note: You need Python 2.4 or above to use generators, you’ll probably have this anyway, but it might pay to check before wondering why this stuff doesn’t work.

A Quick Introduction

At it’s heart a generator is a function which can return a value several times within it’s body. Each time the function is called it resumes from where it terminated in the previous call, with all it’s internal state intact (i.e. variables are held between calls). In Python a generator is defined by creating a function which uses the ‘yield’ keyword. Each time that a yield is hit the function will return the specified value. Let’s start with an example which defines a generator implementation of the built-in ‘range’ function:

def genrange(start, end):
    i = start
    while i < end:
        yield i
        i += 1

As you can see this is incredibly simple and behaves in the same way as the standard range function. The only difference is that when the generator method is called it will return a generator object rather than a value (in contrast to range, which will return a list of values in the range). These generator objects are iteratable so calling the ‘next’ method on the object will run your code and return the next yielded value. Because the generator object is iteratable, the way in which you are likely to use it should look familiar:

for i in genrange(0, 10):
    print i

If you want to go ahead and create a list from the sequence as you generate it, you can use a list comprehension, like so:

l = [i for i in genrange(0, 10)]

Of course, as hist comprehensions can be much more complicated than this you can use it to achieve more than just creating the equivalent list as calling the built-in range function. For example, you could filter the output elements according to some criteria, or call a method on each element returned.

A Trade Off

Although the above example is incredibly simple, it presents an interesting trade off between memory use and the speed of iteration through the ‘list’. The trade off exists because you could just build the list of elements in a separate function and then iterate through it. This is a perfectly valid approach but if there are a large number of elements you will use a significant quantity of memory to store the list. You are also more likely to double handle the data, unless you are building the list up over a period of time. Using a generator in this case is likely to be more efficient, as you are just creating the elements as you need them, so only storing a minimal amount of data at any one time.

The other side of the trade off is performance. If the process that produces your iterative data is computationally expensive, it makes no sense to calculate more values than you will need. So if you have a data processing loop, that can potentially exit early (i.e. before reaching the end of the ‘list’) a generator will be more efficient. But equally, if you require a tight loop which iterates quickly, you would want to pre-compute the values.

When considering using generators you should weigh up these factors and think which method is going to be most efficient for your situation. I think in a lot of cases, you will end up coming down in favour of the generator solution.

Another Example (Dan Brown is going to love me)

I’m going to leave you with another quick example, namely a very efficient way of generating Fibonacci numbers using a generator. Without further ado:

def fibonacci():
    p1 = 0
    p2 = 1
    while True:
        f = p1 + p2
        p2 = p1
        p1 = f
        yield f

There you go, easy. Many other Fibonacci implementations use a recursive method, which is horribly inefficient. Most other implementations will require either a class to store persistent data in, or the use of static variables, both of which are more tricky to handle from a programming point of view.

Well that’s it. I think I’ve demonstrated that generators are pretty cool. The examples I’ve presented are very simplistic, but I’m sure you can think of awesome uses for them. Go forth and code!

The main piece of software you will need is dnsmasq, which you can install with the command:

sudo apt-get install dnsmasq

I already had this installed as I’m using it as a DNS cache for my network (which is also pretty useful). It turns out that dnsmasq is a bit of a ‘swiss army knife’, it can do DHCP, DNS and TFTP all together and very easily. You should edit the config file:

sudo nano /etc/dnsmasq.conf

And add/uncomment/edit the following lines:

dhcp-range=192.168.1.3,192.168.1.50,12h # sets the ip address range and update frequency for the network
dhcp-boot=pxelinux.0 # set up network booting to boot the pxelinux bootloader
enable-tftp # enable built in tftp server
tftp-root=/var/tftpd # set tftp directory

Basically this sets up dnsmasq to be both the DHCP server for the network and push the PXE boot options out over the network. It also enables the built in TFTP server in dnsmasq and sets the directory to serve files from.

The next step is to get some files to serve, I got the PXE/netboot image from the Ubuntu cdimages site, and untarred the files to the correct directory:

wget http://archive.ubuntu.com/ubuntu/dists/jaunty/main/installer-i386/current/images/netboot/netboot.tar.gz
sudo mkdir -p /var/tftp
tar -xvzf netboot.tar.gz -C /var/tftp/
chown -R nobody:nogroup /var/tftp

The tar file of course is basically just the minimal install CD, set up for netbooting, so when you install you will need to download all the packages, for a simple cli install this doesn’t take very long, and you can use a package cache or something to help with future installs.

OK, now restart dnsmasq:

sudo /etc/init.d/dnsmasq restart

…and you should be good to go (make sure that your machine is the only DHCP server on the network). For tips on booting clients see my original article – btw, since I used the new version of Ubuntu I can now boot my Eee PC 901 over the network!

If you have any questions or problems with this I’d be happy to answer them, just post them below.

Bye for now!