Blogging and Development Haitus

People following me via and via the site feed may have noticed that I’ve been a bit quiet lately. I won’t go into the details about what’s been going on (its all been good stuff though), but its safe to say I’ve been a bit busy. Anyway, I’m back (pretty much).

I’ve decided now to focus my spare time on SwallowCatcher development and poking around with cool technologies that I come across, with some write-ups here. This means that I’m ditching the Unofficial Python Module of the Week segment. This is really because I just can’t hack the schedule and don’t want to be tied down to blogging at specific times.

However, on the up side SwallowCatcher development will resume from today and I hope to have a new release soon. Of course the new release will be posted here as soon as it’s ready.

Also, apologies to anyone who came across my site in its eye-offending pink and grey glory. I’m not sure what happened there. I’m hoping that the site wasn’t hacked in my absence and I can’t find any other evidence of that, but its weird nonetheless.

Unofficial Python Module of the Week #2: configobj

Welcome to our third instalment of interesting Python modules. Unfortunately I’m a bit late with this section this week – in fact its next week already! The fourth instalment should be along towards the end of the week thus catching me up.

Today we’re going to cover something which isn’t in the standard library, but is nonetheless very useful. The module is configobj which is used for reading from and writing to INI style configuration files files. A simple INI file is shown below:

item1 = value
item2 = value2

[ section1 ]
item1 = value

[[ subsection ]]
item1 = value

In the above we can see the simple use of items, values sections and subsection. Subsections can be nested down as far a you want, but I don’t think most applications will need many more than two or three levels.


As this module isn’t in the standard library, we need to install it. On most Linux distros it should be in the package repositories, for example on Fedora 14:

$ sudo yum install python-configobj

Windows and Mac users can install from PyPi by following the instructions on the homepage.

Basic Usage

Reading from a configuration file with configobj couldn’t really be any simpler:

import configobj
config = configob.ConfigObj(filename)
myoption = config['item1']
mysectionoption = config['section1']['item1']
mysubsectionoption = config['section1']['subsection']['item1']

Basically, all you need to do is open a ConfigObj object by passing it a filename, then you just read from it as if its a dictionary object. Sections and subsections appear as nested dictionaries. Writing to the file is just as simple:

import configobj
config = configob.ConfigObj(filename)
config['newoption'] = 'new stuff'

No surprises here, you just write to it as if it were a dictionary. All you have to do it call the write() method when you’ve finished, in order to sync everything to disk.

That’s pretty much it for basic usage. There is much more you can do with configobj, including advanced stuff like validation of configuration files. Check out the documentation for more info.

Unofficial Python Module of the Week #1: shelve

Here we are, the second Unofficial Python Module of the Week. Yes, the second – we started from zero (obviously!). This week we are covering the shelve module. Shelve provides you with a very simple Python object store. You can use it where you need quick persistent storage of objects between program runs, it’s much less overhead than using a database – even SQLite. Anyway, lets dive straight into it:

>>> import shelve
>>> shelf ="myshelf.db", writeback=True)

Here we import the shelve module (its in the standard library, so there’s no installation required). Then we open our persistent object store, supplying the filename that we want to store the objects in and the writeback parameter, which allows mutable objects to be stored more conveniently (otherwise they are only written when an assignment is performed). The writeback parameter also causes data to be cached in memory, which can be quite memory intensive, so you should call shelf.sync() every so on to flush everything to disk.

You can store anything that can be handled by the Python pickle module in a shelf:

>>> shelf['thedict'] = {'one': 1, 'two': 2, 'three': 3}
>>> shelf.sync()

As you can see, using a shelf is just like using a dictionary. The only real limit is that the keys must be strings. You can also read back values from the shelf as with a dictionary:

>>> print(shelf['thedict'])
{'one': 1, 'two': 2, 'three': 3}

That’s just about it, just remember to close the shelf when you’re finished with it:

>>> shelf.close()

If you want to find out more have a look at the official Python docs for shelve and Doug Hellmann’s PyMOTW posting on the subject.

Unofficial Python Module of the Week #0: argparse

[EDIT: This series has been renamed so as not to conflict with Doug Hellman’s excellent Python Module of the Week series. See UPMotW #1 for details]

I’m going to try something different today. This is going to be the first in a series (hopefully once weekly) of Python Modules of the Week. This has probably been done to death elsewhere, but I’m using it as an opportunity to learn some more Python and to cement some of these modules in my mind.

Argparse is a Python module for processing command line arguments. It’s new in version 2.7 and is designed to replace the Optparse module. I’m not sure what the rationale for the change was as the two modules look very similar to me. However, it suffices to say that you should use argparse in all new code.

The main class of argparse is called ArgumentParser, it allows you to add arguments to your program and parse the provided arguments at runtime:

parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose") # a flag type argument
parser.add_argument("value", type=int) # an integer positional argument
args = parser.parse_args() # parse the args from sys.argv
print(args.value) # arguments accessed as data members

You can also add meta-data such as help and descriptions to your arguments via keyword arguments to many of the methods. Take a look at the module documentation for examples. Once nice thing is that when you provide this data, argparse automatically generates the -h and –help flags and populates them with nicely formatted help output using the info you provide.

The argument parsing provided by argparse allows you to create some very complex command line interfaces to your scripts. I especially like the sub-commands framework, which allows each sub-command to exist as its own separate entity with its own options and help information. To create an argument parser with a sub-command is just a few lines:

parser = argparse.ArgumentParser()
subcommands = parser.add_subparsers()
mysubcommand = subcommands.add_parser('mysubcommand')

Of course,each sub-command is an ArgumentParser in its own right, so you can add all the options that you normally would, including command line flags, positional arguments and help information. Also each sub-command will be listed in the help and will also have its own help page which can be accessed via:

./ subcommand --help

All in all, this is a very useful Python module to know about. Its ease of use and the number of things it ‘just takes care of’ make it a pleasure to use. Hopefully it’ll have you writing nice command line interfaces to all those hacked together scripts that are currently controlled by commenting bits of them in and out!