mqtt

Zigbee sniffer dongle for zigbee2mqtt

zigbee2mqtt: Cheap Zigbee Without a Gateway

1 Comment

This post may contain affiliate links. Please see the disclaimer for more information.

I’m continuing to work through my backlog of completed but as yet undocumented Home Automation projects. As such you may be familiar with zigbee2mqtt already, since it’s been around for a while. However, I wanted to document my setup and show off a few automations for some specific use cases of mine.

Introduction

Two sensor projects have constantly eluded me in my ongoing efforts to transform my house into a DIY smarthome. The first of these are open/closed sensors for both internal and external doors and all the windows. The second are some input buttons allowing manual switching of devices in the cases where things aren’t automatically set correctly.

I’ve followed the various options over time and never quite managed to find anything I was happy with. I’ve even strongly considered building my own solution (as I’m doing with my room sensors). However, the trifecta of low power battery operation, nice compact physical design and being cheap enough to install everywhere stumped me. Then the Xiaomi Aqara/Mijia range of sensors started appearing. Those looked perfect, except for that horrible gateway. Enter zigbee2mqtt!

Zigbee2mqtt is a nodejs based software project that uses a cheap zigbee sniffer module with some custom firmware to replace the manufacturers gateways for a range of zigbee devices. As well as the Xiaomi devices it also supports Philips Hue and IKEA Tradfri devices among others. The software connects to the USB serial port provided by the dongle and sends messages via MQTT. It even comes with built in Home Assistant MQTT discovery support! (regular readers will know how much of a fan I am of that).

A Setup for Testing

To start playing around with these devices and not really knowing how well they were going to work in general and specifically with zigbee2mqtt, I decided to start with just a few devices. To this end, I ordered:

In addition, I needed the a CC-debugger programming tool and the associated cable in order to program the CC2531 dongle.

Installation and Setup

Once these arrived I was able to easily program the custom firmware to the CC2531 using the official instructions (I used the Linux version). After that I installed zigbee2mqtt again following the instructions. The only wrinkle in this process was getting the USB serial device to show up in the LXD container that runs my home automation components. This is reasonably specific to my environment of running in LXD, you wouldn’t encounter it on a bare Linux system.

Zigbee sniffer dongle for zigbee2mqtt
The zigbee sniffer dongle installed in the server. The green LED is extremely bright!

After that I set about pairing my devices, again following the excellent documentation. I found that I had to have my devices very close (within 30cm) to the dongle during the pairing process. I’m not familiar with the specifics of the Zigbee pairing process. However, I’m assuming that it uses a very low transmit power. This would act as a security measure to prevent pairing with unauthorised devices. Once the devices were paired I was immediately able to see the door sensors as binary sensors in HASS.

I was able to complete this whole process in about an hour. It was really almost too easy. Huzzah for more than adequate documentation and things going according to plan (that almost never happens).

Reading Button Presses

Integrating the buttons into Home Assistant is a little more difficult. This is because there is no built in abstraction for a button or control surface. Instead we fall back to catching the value published via MQTT when the button is pressed (actually, I note that there is a way to do this using an entity now, but that didn’t exist at the time I did it). Here is the set of automations for the button I put in my kitchen:

Phew, that was ALOT of YAML! Each of these automations is basically the same, with a trigger on the relevant topic from MQTT. There then follows some conditions. The most important of these uses a template to unpack the click field of the JSON payload and check it against the type we are interested in. In this way we can do different things depending on the click type (e.g. single, double and triple clicks or long presses).

The other conditions in the automations perform some state checking to make sure that the action performed is correct depending on the current state of the house. For example there is no point turning the kitchen downlights on when they are already on. Instead the user clearly wants to turn them off. I spent quite some time tweaking these conditions over a few days of usage to get them just right. The final (and easiest) part of each automation is just performing the required action (or actions).

Button Functionality

With all this in place we end up with a single button which has the following functionality:

  • On a single click turn on or off the kitchen downlights, depending on their state at the time
  • For a double click turn on or off the dining room spotlights, again depending on their state
  • On a long click turn on or off all lights in our main living space, again depending on their state
  • A triple press turns off all the lights except the dining room which get dimmed (our romantic dinner scene)

That doesn’t even exhaust the capabilities of this switch. It can do quadruple (and maybe quintuple?) presses as well as variable length long presses. At some point the usability just starts to get silly though.

Xiaomi button driven by zigbee2mqtt
I installed one of the buttons in the kitchen, above the other light switches.

I have the other button set up to do something similar. However it’s on;y switching two things so it’s a little less interesting than the example above.

Monitoring Battery Status

Since these are battery powered devices, it’s useful to be able to monitor the battery level. In doing so we can avoid the otherwise inevitable situation of a significant other pressing the button and the lights not coming on. Luckily, zigbee2mqtt has us covered in this respect. For my buttons and sensors I added the following sensors for my battery levels:

Here I’m just showing two sensors, one for the battery level of the door sensor and one for the battery level of the button. I obviously have similar configuration entries for the other devices.

For some reason the configuration required is a little inconsistent here. I found this was because the button has an entity created via MQTT discovery, which includes the battery level as an attribute, whilst the battery level is not included in the attributes for the door sensor entity.

This means that we create a raw MQTT sensor for the door sensor battery level and extract the reading out in the value_template. For the button battery sensor, we use a template sensor and just pull the required attribute out.

Battery Notifications

Of course, just having the battery levels as entities in Home Assistant is of limited use. What we really want it a notification when the battery gets low:

Here we are triggering when any of the battery sensors we are interested in get below 20%. We then send the notification via the default notification service, using the trigger template variable to include the name of the sensor that triggered the alert and the state.

I actually have two such rules in my set up. This one at 20% and another critical warning at 5%. In this way I get an early heads up of a low battery in order to check I have replacement batteries ready. I then get another when the battery actually needs changing.

Quick Bonus Automation

As I haven’t said much about the actual door sensors in this post, here are the automations I use to turn on and off a dimmed downlight in the kitchen if the door opens at night:

These are pretty self explanatory so I won’t go into them further. The interesting thing here is that the combined reaction time of the sensors, HASS server and smart bulb is so quick that the light is always on by the time the door is open enough to see the bulb. More than once this has tricked me into thinking that the off automation is not actually working and that the light is staying on all night. I found myself waiting outside the door a few times for the light to go off, until I had built up confidence that it was working as expected.

Xiaomi door sensor driven by zigbee2mqtt
One of the door sensors installed on the door. Due to the design of our doors I opted to install the sensor on the inside of the door jamb with the magnet at 90 degrees, which works perfectly.

Impressions of the System

I’m really happy with both the software and hardware components of this project. Aside from the inevitable tweaking of the automation rules I had all four devices that I ordered working in an evening. This included flashing the dongle, installing zigbee2mqtt and all the HASS configuration and automations. Zigbee2mqtt seems to be a pretty well put together piece of software with excellent documentation. I’ve also had no stability issues from it in the several months it’s been running.

In terms of the sensors themselves, they are physically quite nice and well designed. Although they are made of plastic, the matte finish on the outside makes them feel more expensive than they are. They are also paintable, if you are so inclined. The adhesive tape used to stick the devices to the wall is also incredibly strong. You really have to make sure you get them in the right place before you push down!

Conclusion

Overall, I can recommend both zigbee2mqtt and these specific devices for anyone wanting to introduce zigbee devices to their home automation system. The barrier to entry both in terms of cost and set up time/complexity is low if you already have a bit of Linux and Home Assistant knowledge. For example, if you’ve been running a HASS server for a little while and have tacked a few other integration projects you should have no problem. At this stage I wouldn’t recommend it for absolute newcomers due to the firmware flashing part.

For my part, I will definitely be buying more of the window/door sensors with the aim of fitting out the whole house eventually. I’ll probably also get a couple more buttons too! In addition I’ll be investigating the ever growing list of zigbee2mqtt supported devices to see what else I can add to my system over time.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

HDMI CEC Flow

HDMI CEC for Home Assistant with Node-RED

1 Comment

I set out on a Sunday morning thinking this was going to be a quick project and, not having decided on a blog topic for this week, it seemed like the ideal candidate. I was wrong – about it being a quick project, hopefully not about it being a reasonable subject for a blog post.

This post is brought to you by issue #12846 in Home Assistant (and the letter ‘C’). That is to say, one of my automations was broken by this issue, which has been sitting open on GitHub since the beginning of march with no progress. I don’t want this to sound like the usual “user of Open Source application complains about free stuff”, because I’m not actually complaining. I understand that software breaks and sometimes there aren’t the resources available to fix it. The solution to this is to get more developers paid to work on Free and Open Source Software (but that’s entirely a discussion for outside of this post).

Actually, this post is here to offer a solution (or at least a temporary one) to the issue, outside of Home Assistant, since I couldn’t fix it myself (I took a look at the code in question and I couldn’t work it out – it needs to be done by someone with more familiarity with the Home Assistant core).

My solution is to use Node-RED along with the HDMI CEC nodes to create an auto-discovered MQTT switch with which I can turn on and off my TV. So, let’s get into the flow…

The Flow

HDMI CEC Flow

The HDMI CEC Switch Flow

This flow runs on an instance of Node-RED running on my OSMC based Raspberry Pi sitting behind my TV (for those keeping up at home, this makes two NR instances on my network – so far). Currently, this is the only flow running on this instance, but I’m considering what else I can run now that I have Node-RED available there. I installed Node-RED on OSMC using the official install/upgrade script. I had fully expected installing Node-RED under OSMC to be a major pain, but it turned out to just amount to running that command.

After the install had finished, I created a user for Node-RED since I like it to run under it’s own user and updated the systemd unit file accordingly. I then installed the CEC nodes linked above from the palette manager. Here I ran into a minor bump in the road in that the CEC nodes couldn’t execute the cec-client program. As it turned out the location of that binary is in a weird location on OSMC, so I added the following in the systemd file to set this up:

I also needed to add my new Node-RED user to the video group to allow access to the CEC device:

Where I really got stuck was playing around with the example flow for the CEC nodes. It wasn’t that it didn’t work as advertised, it was that it broke the CEC command passthrough to Kodi running on the same machine, rendering my TV remote useless within Kodi. Many hours, much futile searching and playing with cec-client later, I still wasn’t any closer to a solution. I knew it must work, because somehow the pycec script I was using previously is able to send an receive CEC packets without interfering with Kodi.

The breakthrough was dropping both a CEC-In and a CEC-Out node into my flow and only grabbing a few CEC packet types in the filter of the input node. I say ‘breakthrough’ – this works most of the time, but it throws a few errors and warnings on start up. I found it to be most reliable when I immediately restarted Kodi after deploying it – this also helps Kodi to regain its CEC connection if necessary.

So How Does It Work?

Oh, yeah. I was going to talk about the flow, but I kinda got sidetracked there.

Well, it’s pretty simple there are two sequences in the flow. One which handles the switch state and MQTT discovery configuration (bottom) and one which handles the incoming commands over MQTT and sending the corresponding CEC commands.

Let’s start with the bottom one:

This sequence has two input paths, the bottom of these executes on start up (or at deploy time) and sends the Home Assistant MQTT Discovery configuration, using the same technique I used in my volcano sensors. The start up message also passes through a 3 second delay before passing to an exec node, which restarts Kodi. I added the following to my sudoers file (via visudo), to allow this:

The top input path receives incoming CEC messages of the type REPORT_POWER_STATUS. In my setup, this only receives power messages from the TV, but you may receive messages from other devices on the bus, in this case you can add a check on the source address of the packet in the following function node (clue: the TV is usually address 0).

The message passes through a function node, which converts the power status to the switch status expected by HASS and also sets the MQTT topic:

Both input paths are connected to a common MQTT output node to send their respective messages (config and state) out to Home Assistant.

The top sequence simply subscribes to the command topic from HASS and determines whether the command was on or off. The JSON payload for the CEC command is then set respectively in either branch – this JSON is taken directly from the example flow linked above. Then we pass this out to the CEC adapter – done. When the device acts upon the CEC command it should send its new power state back through and update the state of our switch. The state will also be updated if you turn on the TV by other means, e.g. the remote.

Pure JSON

This JSON was made in clean, green New Zealand from 100% natural ingredients (electrons):

Bonus: Home Assistant Automation Rules

Here are the Home Assistant automations that I’m using with this. Basically I’m turning off the TV five minutes after either Kodi or the Chromecast stops playing, unless it started again in the meantime:

This uses a timer, which is defined as:

Done. Now we can be lazy/forgetful about leaving TV on and also not waste power. Mission Accomplished.

Conclusion

Hopefully, someone will find time to fix the bug above. I’m probably going to stick with this regardless because I had some other issues running pyCEC on top of OSMC – mainly because they don’t build the libcec bindings for Python 3 by default. I had some custom patches to do this, but it would break (in one way or another on every update). Hopefully, this solution should be more robust. Also, the MQTT connection used in this solution runs over TLS (rather than the unencrypted TCP of the pyCEC network mode), so there is a little security win. Plus, as I already mentioned, now I have a Node-RED instance on a Pi in my living room.

Tiny MQTT Broker with OpenWRT

5 Comments

This post may contain affiliate links. Please see the disclaimer for more information.

So yet again I’ve been really lax at posting, but meh. I’ve still been working on various projects aimed at home automation – this post is a taster of where I’m going…

MQTT (for those that haven’t heard about it) is a real time, lightweight, publish/subscribe protocol for telemetry based applications (i.e. sensors). It’s been described as “RSS for the Internet of Things” (a rather poor description in my opinion).

The central part of MQTT is the broker: clients connect to brokers in order to publish data and receive data in feeds to which they are subscribed. Multiple brokers can be fused together in a heirarchical structure, much like the mounting of filesystems in a unix-like system.

I’ve been considering using MQTT for the communication medium in my planned home automation/sensor network projects. I wanted to set up a heirarchical system with different brokers for different areas of the house, but hadn’t settled on a hardware platform. Until now…

…enter the TP-Link MR3020 ‘travel router’, which is much like the TL-WR703N which I’ve seen used in several hardware hacks recently:

It's a Tiny MQTT Broker!

It’s a Tiny MQTT Broker!

I had to ask a friend in Hong Kong to send me a couple of these (they aren’t available in NZ) – thanks Tony! (UPDATE 2019: Of course now you can get these shipped direct, something I didn’t know about in 2012). Once I received them installing OpenWRT was easy (basically just upload through the exisiting web UI, follow the instructions on the wiki page I linked to above). I then configured the wireless adapter in station mode so that it would connect to my existing wireless network and added a cheap 8GB flash drive to expand the available storage (the device only has 4MB of on-board flash, of which ~900KB is available after installing OpenWRT). I followed the OpenWRT USB storage howto for this and to my relief found that the on-board flash had enough space for the required drivers (phew!).

Once the hardware type stuff was sorted with the USB partitioned (1GB swap, 7GB /opt) and mounting on boot, I was able to install Mosquitto, the Open Source MQTT broker with the following command:

$ opkg install mosquitto -d opt

The -d option allows the package manager to install to a different destination, in this case /opt. Destinations are configured in /etc/opkg.conf.

It took a little bit of fiddling to get mosquitto to start at boot, mainly because of the custom install location. In the end I just edited the paths in /opt/etc/init.d/mosquitto to point to the correct locations (I changed the APP and CONF settings). I then symlinked the script to /etc/rc.d/S50mosquitto to start it at boot.

That’s about as far as I’ve got, apart from doing a quick test with mosquitto_pub/mosquitto_sub to test everything works. I haven’t tried mounting the broker under the master broker running on my home server yet.

The next job is to investigate the serial port on the device in order to attach an Arduino clone which I soldered up a while ago. That will be a story for another day, hopefully in the not-to-distant future!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.