Tiny MQTT Broker with OpenWRT

So yet again I’ve been really lax at posting, but meh. I’ve still been working on various projects aimed at home automation – this post is a taster of where I’m going…

MQTT (for those that haven’t heard about it) is a real time, lightweight, publish/subscribe protocol for telemetry based applications (i.e. sensors). It’s been described as “RSS for the Internet of Things” (a rather poor description in my opinion).

The central part of MQTT is the broker: clients connect to brokers in order to publish data and receive data in feeds to which they are subscribed. Multiple brokers can be fused together in a heirarchical structure, much like the mounting of filesystems in a unix-like system.

I’ve been considering using MQTT for the communication medium in my planned home automation/sensor network projects. I wanted to set up a heirarchical system with different brokers for different areas of the house, but hadn’t settled on a hardware platform. Until now…

…enter the TP-Link MR3020 ‘travel router’, which is much like the TL-WR703N which I’ve seen used in several hardware hacks recently:

It's a Tiny MQTT Broker!

It’s a Tiny MQTT Broker!

I had to ask a friend in Hong Kong to send me a couple of these (they aren’t available in NZ) – thanks Tony! Once I received them installing OpenWRT was easy (basically just upload through the exisiting web UI, follow the instructions on the wiki page I linked to above). I then configured the wireless adapter in station mode so that it would connect to my existing wireless network and added a cheap 8GB flash drive to expand the available storage (the device only has 4MB of on-board flash, of which ~900KB is available after installing OpenWRT). I followed the OpenWRT USB storage howto for this and to my relief found that the on-board flash had enough space for the required drivers (phew!).

Once the hardware type stuff was sorted with the USB partitioned (1GB swap, 7GB /opt) and mounting on boot, I was able to install Mosquitto, the Open Source MQTT broker with the following command:

$ opkg install mosquitto -d opt

The -d option allows the package manager to install to a different destination, in this case /opt. Destinations are configured in /etc/opkg.conf.

It took a little bit of fiddling to get mosquitto to start at boot, mainly because of the custom install location. In the end I just edited the paths in /opt/etc/init.d/mosquitto to point to the correct locations (I changed the APP and CONF settings). I then symlinked the script to /etc/rc.d/S50mosquitto to start it at boot.

That’s about as far as I’ve got, apart from doing a quick test with mosquitto_pub/mosquitto_sub to test everything works. I haven’t tried mounting the broker under the master broker running on my home server yet.

The next job is to investigate the serial port on the device in order to attach an Arduino clone which I soldered up a while ago. That will be a story for another day, hopefully in the not-to-distant future!

Installing and Configuring Arch Linux: Part 1

OTHERWISE ENTILED: Rob tries to install Arch Linux some of the time, but really spends most of the time drinking beer.

Before I start: NO, UNLIKE EVERY OTHER ARTICLE ON THE WEB, PUBLISHED TODAY, THIS IS NOT A JOKE, K?!?

I’ve been looking for a new distro recently. I do this from time to time, principally because I get bored of what I’m currently running. Last time it was Crunchbang which I settled on. This time I wanted to go more advanced, so I started researching Arch Linux.

For those that don’t know, Arch Linux describes itself as:

…a lightweight and flexible Linux┬« distribution that tries to Keep It Simple.

I’d heard about Arch in the past from several sources and had heard that you basically have to install and configure everything yourself, but that the package manager (awesomely named Pacman!) manages software without having to compile from source (unless you want to!).

The following series of posts will be a record of my experiences installing and configuring Arch on my home desktop machine. This isn’t intended to be an exhaustive installation guide, more just a record of where I tripped up in order to aid those who come next. If you are searching for an installation guide, try the excellent article on the Arch Wiki.

I’ve separated the post out into days. Note: it didn’t actually take me a full day for each part, I work during the day and only really had a couple of hours each evening to spend on this.

Day 1: Backing Up

Before installing I wanted to make sure I didn’t trash my existing Ubuntu system and all my personal data, as I still need to do all the stuff I usually do with my machine. So I made a backup.

I’m not really going to go into how. Suffice to say I used LVM snapshots and rsync, I might write about this in a future post.

This took a while, as I have quite a lot of data. I thought it best to have a beer in the mean time, so I did.

Day 2: Making Space, Starting the Installation and Various Adventures with LVM

The next thing to do was to resize my existing LVM partition containing Ubuntu so that I had space for Arch. I couldn’t work out how to do this at first as none of the partition tools I tried (GParted and Cfdisk) could resize the partition. I eventually worked out how to do it.

First, on my running Ubuntu system I resized the physical volume with:

$ pvresize --setphysicalvolumesize 500G /dev/sda1

This shrank the space used by LVM down to 500GB (from about 1000GB on my machine).

I then rebooted into the Arch live CD (64-bit edition in my case), and ran:

$ fdisk /dev/sda

Now what you have to do next is slightly alarming. You actually have to delete the partition and recreate it in the new size. This works, without destroying your data, because fdisk only manipulates the partition table on the disk, it doesn’t do any formatting of partitions, etc.

I did this through fdisk so that the partition was 501GB (making it a little bigger than the PV just to make sure). I then rebooted back into Ubuntu and ran:

$ pvresize /dev/sda1

To allow it to use all the space. This probably isn’t necessary but I wanted to be safe.

Next, I proceeded to the installation. For some reason the Arch boot CD was really slow to boot and gave me loads of read errors, I think this might have something to do with my drive as I’ve been experiencing the same with other disks. Eventually it booted and dropped my at the default prompt.

From then I basically followed the installation guide for setting up the package source (CD) and the date and time.

I then set about partitioning the disks. The Arch installer uses Cfdisk, which is fine. I just added two partitions to my disk, a small (255Meg) one for my /boot partition and a large LVM one for the rest of the system (I like LVM and wanted to use it again on Arch).

This was fine, but I had some problems setting up the LVM through the installer, even though the user guide seems to think it can do it. Every time I tried, it would just fail on creating the Volume Group, weird.

I gave up for the evening and (you guessed it) went for a beer!

Day 3: Successful Installation

The next day I thought I’d try googling for LVM on Arch, luckily when I got in to work @duffkitty on identi.ca had seen one of my posts complaining about having problems and had given me a link to the LVM article on the Arch Wiki.

This advocated setting up the whole LVM setup manually (and guides you through it) and then just setting the partitions to use in the installer. It also gives you some important things to look out for when configuring the system. Following these instructions worked like a charm and I was able to format everything correctly and install the base system.

I then moved on to configuring the system, following the install guide and taking into account the instructions in the LVM article. Everything went pretty much fine here and I eventually got to installing the bootloader. Here I replaced the Ubuntu Grub version with the one installed by Arch. This left me having to add an entry for Ubuntu, which wasn’t difficult, I just copied the Arch one and changed the partition and file names.

Then it was time to ‘type reboot and pray’ as the Arch installation guide puts it.

So I did.

When I rebooted the bootloader came up with the Arch and Ubuntu entries. I selected Ubuntu just to check everything was OK.

It didn’t work.

Panicking and Swearing Ensued.

I rebooted and selected Arch.

That worked (thankfully).

When it had booted I logged in and opened up the Grub config file again. it turned out I mis-typed the name of the Ubuntu initrd file, that was easily fixed. Rebooting got me safely back to Ubuntu.

So now I have a functioning dual boot between my original Ubuntu install and a very basic Arch install, I think I might need some software there!

But first… beer.

So What’s Next???

Well, firstly I need to get my network connection up and running as I didn’t do that during the install. It’s a Wifi connection over WPA so that’s going to be fun. Then I can start installing software. I’ll probably follow the Beginners Guide on the Wiki (from Part 3). I was also recommended Yaourt by @duffkitty, so I’ll give that a try.

I’ll be continuing to play with Arch over the next few days and reporting my progress in follow up posts here. I’ll also be denting as I go along and you can follow all of these on my #archrob hash tag.

There’ll probably be beer too.

We’ll see how it goes, but eventually I hope to have a system I can use full time.

Bye for now! Happy Easter!

Online Filesystem Resizing with LVM

I use LVM on my main desktop machine. This is awesome because it allows me to dynamically allocate space to partitions as I choose, however I always forget how to do a resize, so I’m going to write it down here. This isn’t going to be a full LVM tutorial (there’s plenty of material out there for that), although maybe that’s an idea for the future.

The following commands will resize an ext2, ext3, or ext4 filesystem running on LVM while it is mounted:

$ sudo lvresize -L +XXG <path to fs device>
$ sudo resize2fs <path to fs device>

In the above command you need to replace XX with the number of GB you want the filesystem to grow by and <path to fs device> by the device node (typically /dev/mapper/something).

An there you have it, done! Obviously there is a huge amount more you can do with the two tools above, take a look at their man pages for more info.

Hopefully this post will save me from having to work out how to do this every time!