Playing with Python Generators

Today I’ve been playing with generators in Python. I have to say they are yet another awesome Python feature! They give you a very cool and efficient way to store some state within a function, this is useful if you’re trying to do something like generating a sequence, which you would otherwise need a class for. I’m going to go over a couple of quick examples here in order to demonstrate what generators are and what you can do with them.

Side note: You need Python 2.4 or above to use generators, you’ll probably have this anyway, but it might pay to check before wondering why this stuff doesn’t work.

A Quick Introduction

At it’s heart a generator is a function which can return a value several times within it’s body. Each time the function is called it resumes from where it terminated in the previous call, with all it’s internal state intact (i.e. variables are held between calls). In Python a generator is defined by creating a function which uses the ‘yield’ keyword. Each time that a yield is hit the function will return the specified value. Let’s start with an example which defines a generator implementation of the built-in ‘range’ function:

def genrange(start, end):
    i = start
    while i < end:
        yield i
        i += 1

As you can see this is incredibly simple and behaves in the same way as the standard range function. The only difference is that when the generator method is called it will return a generator object rather than a value (in contrast to range, which will return a list of values in the range). These generator objects are iteratable so calling the ‘next’ method on the object will run your code and return the next yielded value. Because the generator object is iteratable, the way in which you are likely to use it should look familiar:

for i in genrange(0, 10):
    print i

If you want to go ahead and create a list from the sequence as you generate it, you can use a list comprehension, like so:

l = [i for i in genrange(0, 10)]

Of course, as hist comprehensions can be much more complicated than this you can use it to achieve more than just creating the equivalent list as calling the built-in range function. For example, you could filter the output elements according to some criteria, or call a method on each element returned.

A Trade Off

Although the above example is incredibly simple, it presents an interesting trade off between memory use and the speed of iteration through the ‘list’. The trade off exists because you could just build the list of elements in a separate function and then iterate through it. This is a perfectly valid approach but if there are a large number of elements you will use a significant quantity of memory to store the list. You are also more likely to double handle the data, unless you are building the list up over a period of time. Using a generator in this case is likely to be more efficient, as you are just creating the elements as you need them, so only storing a minimal amount of data at any one time.

The other side of the trade off is performance. If the process that produces your iterative data is computationally expensive, it makes no sense to calculate more values than you will need. So if you have a data processing loop, that can potentially exit early (i.e. before reaching the end of the ‘list’) a generator will be more efficient. But equally, if you require a tight loop which iterates quickly, you would want to pre-compute the values.

When considering using generators you should weigh up these factors and think which method is going to be most efficient for your situation. I think in a lot of cases, you will end up coming down in favour of the generator solution.

Another Example (Dan Brown is going to love me)

I’m going to leave you with another quick example, namely a very efficient way of generating Fibonacci numbers using a generator. Without further ado:

def fibonacci():
    p1 = 0
    p2 = 1
    while True:
        f = p1 + p2
        p2 = p1
        p1 = f
        yield f

There you go, easy. Many other Fibonacci implementations use a recursive method, which is horribly inefficient. Most other implementations will require either a class to store persistent data in, or the use of static variables, both of which are more tricky to handle from a programming point of view.

Well that’s it. I think I’ve demonstrated that generators are pretty cool. The examples I’ve presented are very simplistic, but I’m sure you can think of awesome uses for them. Go forth and code!

Denting via Android Intents

This is just a quick post to share something cool that I was playing with the other day. I was wondering if it was possible to send a message from an Android application via the Mustard identi.ca client. It turns out it is, and here’s how you do it:

Intent i = new Intent(Intent.ACTION_SEND);
i.setType("text/plain");
i.putExtra(Intent.EXTRA_TEXT, "Testing intents on android");
startActivity(i);

These four lines of code hook into a key feature of the Android platform: Intents. The intent system is a neat way of communicating via apps and enables different apps to integrate together without knowing about each other specifically.

In this case, we are sending the ACTION_SEND intent, which Mustard is set up to handle. We set the mime-type of the message to ‘text/plain’ and put the contents of our message into the EXTRA_TEXT field. We then start the intent, and we’re done!

When the intent is started Android presents the user with a list of all the possible applications which can handle the ACTION_SEND intent. In my case this is Email, Gmail, Messaging and Mustard, when I select mustard the new message screen pops up and is populated with my message content. Hitting send posts the message and returns control to my app. Neat!

There are probably a whole host of ways you can integrate your apps with other apps on the phone using the intents system, I guess it’s just a matter of knowing what intents are received by each app and what they do.

Thanks to @macno (main developer of Mustard) and @bugabundo on identi.ca for their help working this out!

Anyway, I hope someone finds this useful. This is the first time I’ve posted about Android development, but I’ll hopefully write more about it in the future. Bye for now!

Installing and Configuring Arch Linux: Part 1

OTHERWISE ENTILED: Rob tries to install Arch Linux some of the time, but really spends most of the time drinking beer.

Before I start: NO, UNLIKE EVERY OTHER ARTICLE ON THE WEB, PUBLISHED TODAY, THIS IS NOT A JOKE, K?!?

I’ve been looking for a new distro recently. I do this from time to time, principally because I get bored of what I’m currently running. Last time it was Crunchbang which I settled on. This time I wanted to go more advanced, so I started researching Arch Linux.

For those that don’t know, Arch Linux describes itself as:

…a lightweight and flexible Linux® distribution that tries to Keep It Simple.

I’d heard about Arch in the past from several sources and had heard that you basically have to install and configure everything yourself, but that the package manager (awesomely named Pacman!) manages software without having to compile from source (unless you want to!).

The following series of posts will be a record of my experiences installing and configuring Arch on my home desktop machine. This isn’t intended to be an exhaustive installation guide, more just a record of where I tripped up in order to aid those who come next. If you are searching for an installation guide, try the excellent article on the Arch Wiki.

I’ve separated the post out into days. Note: it didn’t actually take me a full day for each part, I work during the day and only really had a couple of hours each evening to spend on this.

Day 1: Backing Up

Before installing I wanted to make sure I didn’t trash my existing Ubuntu system and all my personal data, as I still need to do all the stuff I usually do with my machine. So I made a backup.

I’m not really going to go into how. Suffice to say I used LVM snapshots and rsync, I might write about this in a future post.

This took a while, as I have quite a lot of data. I thought it best to have a beer in the mean time, so I did.

Day 2: Making Space, Starting the Installation and Various Adventures with LVM

The next thing to do was to resize my existing LVM partition containing Ubuntu so that I had space for Arch. I couldn’t work out how to do this at first as none of the partition tools I tried (GParted and Cfdisk) could resize the partition. I eventually worked out how to do it.

First, on my running Ubuntu system I resized the physical volume with:

$ pvresize --setphysicalvolumesize 500G /dev/sda1

This shrank the space used by LVM down to 500GB (from about 1000GB on my machine).

I then rebooted into the Arch live CD (64-bit edition in my case), and ran:

$ fdisk /dev/sda

Now what you have to do next is slightly alarming. You actually have to delete the partition and recreate it in the new size. This works, without destroying your data, because fdisk only manipulates the partition table on the disk, it doesn’t do any formatting of partitions, etc.

I did this through fdisk so that the partition was 501GB (making it a little bigger than the PV just to make sure). I then rebooted back into Ubuntu and ran:

$ pvresize /dev/sda1

To allow it to use all the space. This probably isn’t necessary but I wanted to be safe.

Next, I proceeded to the installation. For some reason the Arch boot CD was really slow to boot and gave me loads of read errors, I think this might have something to do with my drive as I’ve been experiencing the same with other disks. Eventually it booted and dropped my at the default prompt.

From then I basically followed the installation guide for setting up the package source (CD) and the date and time.

I then set about partitioning the disks. The Arch installer uses Cfdisk, which is fine. I just added two partitions to my disk, a small (255Meg) one for my /boot partition and a large LVM one for the rest of the system (I like LVM and wanted to use it again on Arch).

This was fine, but I had some problems setting up the LVM through the installer, even though the user guide seems to think it can do it. Every time I tried, it would just fail on creating the Volume Group, weird.

I gave up for the evening and (you guessed it) went for a beer!

Day 3: Successful Installation

The next day I thought I’d try googling for LVM on Arch, luckily when I got in to work @duffkitty on identi.ca had seen one of my posts complaining about having problems and had given me a link to the LVM article on the Arch Wiki.

This advocated setting up the whole LVM setup manually (and guides you through it) and then just setting the partitions to use in the installer. It also gives you some important things to look out for when configuring the system. Following these instructions worked like a charm and I was able to format everything correctly and install the base system.

I then moved on to configuring the system, following the install guide and taking into account the instructions in the LVM article. Everything went pretty much fine here and I eventually got to installing the bootloader. Here I replaced the Ubuntu Grub version with the one installed by Arch. This left me having to add an entry for Ubuntu, which wasn’t difficult, I just copied the Arch one and changed the partition and file names.

Then it was time to ‘type reboot and pray’ as the Arch installation guide puts it.

So I did.

When I rebooted the bootloader came up with the Arch and Ubuntu entries. I selected Ubuntu just to check everything was OK.

It didn’t work.

Panicking and Swearing Ensued.

I rebooted and selected Arch.

That worked (thankfully).

When it had booted I logged in and opened up the Grub config file again. it turned out I mis-typed the name of the Ubuntu initrd file, that was easily fixed. Rebooting got me safely back to Ubuntu.

So now I have a functioning dual boot between my original Ubuntu install and a very basic Arch install, I think I might need some software there!

But first… beer.

So What’s Next???

Well, firstly I need to get my network connection up and running as I didn’t do that during the install. It’s a Wifi connection over WPA so that’s going to be fun. Then I can start installing software. I’ll probably follow the Beginners Guide on the Wiki (from Part 3). I was also recommended Yaourt by @duffkitty, so I’ll give that a try.

I’ll be continuing to play with Arch over the next few days and reporting my progress in follow up posts here. I’ll also be denting as I go along and you can follow all of these on my #archrob hash tag.

There’ll probably be beer too.

We’ll see how it goes, but eventually I hope to have a system I can use full time.

Bye for now! Happy Easter!

Even easier netboot installation…

A while ago I covered netbooting/installation on ubuntu, well I’ve now found an even easier way to do this! It’s probably the easiest way to go about this as it really only involves editing one config file. Some of the info here comes from the official Ubuntu documentation on this, though my approach is actually easier, since you don’t need a separate tftp server.

The main piece of software you will need is dnsmasq, which you can install with the command:

sudo apt-get install dnsmasq

I already had this installed as I’m using it as a DNS cache for my network (which is also pretty useful). It turns out that dnsmasq is a bit of a ‘swiss army knife’, it can do DHCP, DNS and TFTP all together and very easily. You should edit the config file:

sudo nano /etc/dnsmasq.conf Continue reading

Quickly change Debian repositories

Apt is awesome. Plain and simple.

But it is kinda static. By this I mean it’s not particularly suited to environments where things change frequently. For example, we have a local mirror at uni, which of course it much faster than using the external Ubuntu or Debian ones, however as this is only available from internal University of Auckland IP addresses I would have to change my /etc/apt/sources.list file if I wanted to install something from home.

Today I knocked together a quick Python script to fix this, all it does is basically manipulate a symlink which points to the real /etc/apt/sources.list file, but I thought I’d share it anyway: Continue reading