home automation

Zigbee sniffer dongle for zigbee2mqtt

zigbee2mqtt: Cheap Zigbee Without a Gateway

Leave a comment

I’m continuing to work through my backlog of completed but as yet undocumented Home Automation projects. As such you may be familiar with zigbee2mqtt already, since it’s been around for a while. However, I wanted to document my setup and show off a few automations for some specific use cases of mine.

Introduction

Two sensor projects have constantly eluded me in my ongoing efforts to transform my house into a DIY smarthome. The first of these are open/closed sensors for both internal and external doors and all the windows. The second are some input buttons allowing manual switching of devices in the cases where things aren’t automatically set correctly.

I’ve followed the various options over time and never quite managed to find anything I was happy with. I’ve even strongly considered building my own solution (as I’m doing with my room sensors). However, the trifecta of low power battery operation, nice compact physical design and being cheap enough to install everywhere stumped me. Then the Xiaomi Aqara/Mijia range of sensors started appearing. Those looked perfect, except for that horrible gateway. Enter zigbee2mqtt!

Zigbee2mqtt is a nodejs based software project that uses a cheap zigbee sniffer module with some custom firmware to replace the manufacturers gateways for a range of zigbee devices. As well as the Xiaomi devices it also supports Philips Hue and IKEA Tradfri devices among others. The software connects to the USB serial port provided by the dongle and sends messages via MQTT. It even comes with built in Home Assistant MQTT discovery support! (regular readers will know how much of a fan I am of that).

A Setup for Testing

To start playing around with these devices and not really knowing how well they were going to work in general and specifically with zigbee2mqtt, I decided to start with just a few devices. To this end, I ordered:

In addition, I needed the a CC-debugger programming tool and the associated cable in order to program the CC2531 dongle.

Installation and Setup

Once these arrived I was able to easily program the custom firmware to the CC2531 using the official instructions (I used the Linux version). After that I installed zigbee2mqtt again following the instructions. The only wrinkle in this process was getting the USB serial device to show up in the LXD container that runs my home automation components. This is reasonably specific to my environment of running in LXD, you wouldn’t encounter it on a bare Linux system.

Zigbee sniffer dongle for zigbee2mqtt
The zigbee sniffer dongle installed in the server. The green LED is extremely bright!

After that I set about pairing my devices, again following the excellent documentation. I found that I had to have my devices very close (within 30cm) to the dongle during the pairing process. I’m not familiar with the specifics of the Zigbee pairing process. However, I’m assuming that it uses a very low transmit power. This would act as a security measure to prevent pairing with unauthorised devices. Once the devices were paired I was immediately able to see the door sensors as binary sensors in HASS.

I was able to complete this whole process in about an hour. It was really almost too easy. Huzzah for more than adequate documentation and things going according to plan (that almost never happens).

Reading Button Presses

Integrating the buttons into Home Assistant is a little more difficult. This is because there is no built in abstraction for a button or control surface. Instead we fall back to catching the value published via MQTT when the button is pressed (actually, I note that there is a way to do this using an entity now, but that didn’t exist at the time I did it). Here is the set of automations for the button I put in my kitchen:

Phew, that was ALOT of YAML! Each of these automations is basically the same, with a trigger on the relevant topic from MQTT. There then follows some conditions. The most important of these uses a template to unpack the click field of the JSON payload and check it against the type we are interested in. In this way we can do different things depending on the click type (e.g. single, double and triple clicks or long presses).

The other conditions in the automations perform some state checking to make sure that the action performed is correct depending on the current state of the house. For example there is no point turning the kitchen downlights on when they are already on. Instead the user clearly wants to turn them off. I spent quite some time tweaking these conditions over a few days of usage to get them just right. The final (and easiest) part of each automation is just performing the required action (or actions).

Button Functionality

With all this in place we end up with a single button which has the following functionality:

  • On a single click turn on or off the kitchen downlights, depending on their state at the time
  • For a double click turn on or off the dining room spotlights, again depending on their state
  • On a long click turn on or off all lights in our main living space, again depending on their state
  • A triple press turns off all the lights except the dining room which get dimmed (our romantic dinner scene)

That doesn’t even exhaust the capabilities of this switch. It can do quadruple (and maybe quintuple?) presses as well as variable length long presses. At some point the usability just starts to get silly though.

Xiaomi button driven by zigbee2mqtt
I installed one of the buttons in the kitchen, above the other light switches.

I have the other button set up to do something similar. However it’s on;y switching two things so it’s a little less interesting than the example above.

Monitoring Battery Status

Since these are battery powered devices, it’s useful to be able to monitor the battery level. In doing so we can avoid the otherwise inevitable situation of a significant other pressing the button and the lights not coming on. Luckily, zigbee2mqtt has us covered in this respect. For my buttons and sensors I added the following sensors for my battery levels:

Here I’m just showing two sensors, one for the battery level of the door sensor and one for the battery level of the button. I obviously have similar configuration entries for the other devices.

For some reason the configuration required is a little inconsistent here. I found this was because the button has an entity created via MQTT discovery, which includes the battery level as an attribute, whilst the battery level is not included in the attributes for the door sensor entity.

This means that we create a raw MQTT sensor for the door sensor battery level and extract the reading out in the value_template. For the button battery sensor, we use a template sensor and just pull the required attribute out.

Battery Notifications

Of course, just having the battery levels as entities in Home Assistant is of limited use. What we really want it a notification when the battery gets low:

Here we are triggering when any of the battery sensors we are interested in get below 20%. We then send the notification via the default notification service, using the trigger template variable to include the name of the sensor that triggered the alert and the state.

I actually have two such rules in my set up. This one at 20% and another critical warning at 5%. In this way I get an early heads up of a low battery in order to check I have replacement batteries ready. I then get another when the battery actually needs changing.

Quick Bonus Automation

As I haven’t said much about the actual door sensors in this post, here are the automations I use to turn on and off a dimmed downlight in the kitchen if the door opens at night:

These are pretty self explanatory so I won’t go into them further. The interesting thing here is that the combined reaction time of the sensors, HASS server and smart bulb is so quick that the light is always on by the time the door is open enough to see the bulb. More than once this has tricked me into thinking that the off automation is not actually working and that the light is staying on all night. I found myself waiting outside the door a few times for the light to go off, until I had built up confidence that it was working as expected.

Xiaomi door sensor driven by zigbee2mqtt
One of the door sensors installed on the door. Due to the design of our doors I opted to install the sensor on the inside of the door jamb with the magnet at 90 degrees, which works perfectly.

Impressions of the System

I’m really happy with both the software and hardware components of this project. Aside from the inevitable tweaking of the automation rules I had all four devices that I ordered working in an evening. This included flashing the dongle, installing zigbee2mqtt and all the HASS configuration and automations. Zigbee2mqtt seems to be a pretty well put together piece of software with excellent documentation. I’ve also had no stability issues from it in the several months it’s been running.

In terms of the sensors themselves, they are physically quite nice and well designed. Although they are made of plastic, the matte finish on the outside makes them feel more expensive than they are. They are also paintable, if you are so inclined. The adhesive tape used to stick the devices to the wall is also incredibly strong. You really have to make sure you get them in the right place before you push down!

Conclusion

Overall, I can recommend both zigbee2mqtt and these specific devices for anyone wanting to introduce zigbee devices to their home automation system. The barrier to entry both in terms of cost and set up time/complexity is low if you already have a bit of Linux and Home Assistant knowledge. For example, if you’ve been running a HASS server for a little while and have tacked a few other integration projects you should have no problem. At this stage I wouldn’t recommend it for absolute newcomers due to the firmware flashing part.

For my part, I will definitely be buying more of the window/door sensors with the aim of fitting out the whole house eventually. I’ll probably also get a couple more buttons too! In addition I’ll be investigating the ever growing list of zigbee2mqtt supported devices to see what else I can add to my system over time.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter.

Installed room sensor

Room Sensor Project: Part 2 – Infrastructure and Mounting

2 Comments

This post is part of a series on this project. Here is the series so far:

Looking back, it seems to have been a ridiculously long time since my last room sensor post. It’s well over a year, but it doesn’t seem that long ago. This project has been majorly delayed by a few issues and generally hasn’t been top of my todo list. However, I’m now at the point where I have the prototype sensor installed and working. Actually, it’s been working for several months, but I hadn’t got around to writing it up! In this post I’ll mostly be detailing the infrastructure used to get power to the sensor. There will also be some discussion on the case and mounting as well as a few words on the software.

Finding a Case

It would have been really nice just to 3D print a case and mounting bracket for the sensors. Unfortunately, I don’t (yet) have a 3D printer and it was cheaper to buy cases than to get a 3D printing service to print them. I settled on a 100x60x25mm case and ordered 15 of them. Once they arrived I was able to fit all the electronics inside and cut a slot in the bottom for the DHT22 sensor. A dremel-like tool would have helped a lot here, but I managed to do it manually and it looks OK. I actually reversed the case so that the lid became the rear as this looks a little nicer and helped with the mounting of the components.

I also fitted a light sensor based on an LDR voltage divider circuit to the front of the case. Unfortunately I had issues with the ADC pin on the bare ESP8266 module I used for the prototype. It’s odd because I’ve had this working with the Wemos D1 modules before (which have their own voltage divider on the input also). In the end I didn’t manage to get this working and have resolved to replace the prototype board with a Wemos based one when I do the sensors for the rest of the house.

I drilled a hole in the front of the case for the PIR sensor and mounted the diffuser over the hole. The PIR board itself was hot glued to the reverse of this. This works really well – in fact if anything the sensor is a little too sensitive. I need to tweak the pots a little to dial this down.

You can see pictures from during the assembly as well as the fully assembled case below:

Main room sensor board mounted to case lid.
Main room sensor board mounted to case lid.
PIR difuser mounted to case (front view)
PIR diffuser mounted to case (front view)
PIR difuser mounted to case (top view)
PIR diffuser mounted to case (top view)
PIR sensor mounted to inside of case.
PIR sensor mounted to inside of case.
Side view of room sensor board showing power connections
Side view of room sensor board showing power connections
Fully assembled case (upside down)
Fully assembled case (upside down)
Fully assembled case (side view)
Fully assembled case (side view)
Fully assembled case (standing on the DHT sensor)
Fully assembled case (standing on the DHT sensor)

Power Setup and Wiring

In order to get power to the sensors I had decided on running 12V lines through the roof space of the house. These would then come down through the ceilings in the corner of each room for the sensors. The cables would be fed from a central distribution board mounted near the loft hatch in the ceiling. Since the power requirements for the 12 sensors I wish to eventually install are minimal a single 2A is enough to power the whole lot with some room to spare. Some pictures of the distribution board (before and after installation) are shown below:

Room sensor power distribution board (before installation)
Unwired room sensor power distribution board (before installation)
Room sensor power distribution board (post installation)
Room sensor power distribution board (post installation)

I had initially wanted to mount the power supply down in the ‘server rack’ and run the power up through an (existing) whole in the wall to the loft. However, after an abortive attempt at running a cable through the wall (in which I was foiled by pesky insulation and there was much swearing), I eventually came to the conclusion that mains power was needed in the roof space.

Time passes…

It took a while to really commit to and allocate funds to this option. Eventually the electrician came and installed four shiny new power points in the roof space just next to the loft hatch. Four power points is obviously overkill for this project. However, in the intervening time some other projects had come up for which the remaining plugs would be useful.

Once the power points were in place I ran the cabling for the 12V line to the room in which the sensor was to be installed and connected it all up. As if by magic power flowed and the sensor sprung into life! (barring various frustrating issues with loose connections).

I had quite an interesting time working out how to mount the sensor in the corner of the room. I initially stuck it up with 3M double sided sticky pads, but ended up pulling it up and down several times so that I ran out of these. Eventually I opted for good old fashioned blu-tack as a temporary solution! I’m intending to replace this with a 3D printed bracket which will fit the oddly shaped space between the case and the wall. This will allow the sensor to be attached much more permanently. However, for now the blu-tack does the job and proves the concept.

Installed room sensor
Installed room sensor

Relay Power Control

I mentioned above that I had installed several extra power sockets in the roof space for other projects. One of those other projects required putting a Raspberry Pi in the roof space. The other project is now completed and I’m hoping to document it soon. For this I pressed into duty my old Raspberry Pi Model B (the one with 512Mb of RAM). Although old, this hardware is sufficient for running a small Node-RED instance as well as performing it’s other intended duty.

This gave me a nice way to control the power supply to the room sensor distribution board and hence all the sensors. That meant that if a sensor were to go offline I could cycle the power remotely without having to climb up in the roof space. To do this I inserted a relay into the power 12V line between the power supply and the power distribution board.

The Raspberry Pi with the relay assembly
The Raspberry Pi with the relay assembly

I connected this to the normally closed input of the relay so that the relay must be switched on to kill the power. The state is then inverted in Node-RED. In this way the switch in Home Assistant shows as on most of the time. I only had 5V relays sitting in my parts box. So, I soldered up a quick transistor circuit on a breadboard to allow me to drive the relay from the 3.3V logic of the Pi. Doing this is left as an exercise for the reader, since I forgot to document what I built!

Driving the Relay in Node-RED

To drive the relay I use a variation of my MQTT discovery switch in Node-RED. This is implemented via the following flow:

The room sensor relay control flow
The room sensor relay control flow

The JSON for this is shown below (copy it and import into Node-RED):

Over the following months of usage, I noted several (infrequent) instances where the sensor stopped responding. In these cases it needed a manual (though remote controlled) power cycle. In order to automate this and so reduce downtime, I wrote the following Home Assistant automation:

Basically, this will trigger after the sensor has been offline for 10 minutes. Once triggered it will turn the sensor off and on again (with a 30 seconds delay in between). It will also send a notification to inform me that this has happened. I think this has been triggered twice and as a result the sensor hasn’t been unavailable for any length of time.

Software Changes

Since installing the prototype sensor, I haven’t actually been running my Micropython Room Sensor software on it. Instead I’ve been trying out ESPHome on this and another project, since it’s been getting a lot of attention in the HASS community recently. I specifically wanted to see if ESPHome was an easier/maintenance free option for these types of projects.

My take away from this is that ESPHome is really nice and very easy if you don’t want to do anything complicated. If all you have are a few sensors or actuators that you want to connect, it’s great! In fact it’s almost perfect for this kind of project. You can even do some moderately complicated data conversions and on device automation using the lambda syntax. For this reason, I’d put it in the same basket as the likes of ESPeasy. Although it has some advantages in comparison to other systems, especially if you are already running Home Assistant. Kudos to Otto Winter for coming up with such a great piece of software!

However, it does get more difficult when you want to more complicated things. I ran into some of these issues in my other project, which I’ll detail when I eventually write it up. For now ESPHome gets my wholehearted recommendation.

ESPHome Configuration

I especially like that since the configuration for ESPHome devices is just YAML it’s really easy to store in git. I haven’t got a cleaned up git repo for my projects ready to publish. However, since the configuration for this project is so simple, I can post the whole thing here:

You’ll notice that I’m still using the MQTT transport rather that the native API component with the Home Assistant ESPHome component. This is mainly because I built this before the native API was released and I didn’t need to update it. I understand there are some advantages to using the native API, so I probably will try it at some point, especially if I want to try an esp-cam project.

What’s Next

So far, I’m really happy with the performance of the sensor. I’ve been using it in a few automations which I’m intending to detail in a further post. The next thing to do is build further sensors for the remainder of the house. In order to make this less error prone I’ve decided to design an adapter PCB in Kicad for the Wemos D1 Mini clones I’ve been using in other projects. This will get me away from the fiddling with bare ESP modules and hopefully mean that the light sensor will work.

As mentioned above, I also want to design a 3D printed bracket to fit the oddly shaped space in the corner behind the sensor. This will have to wait until I get a 3D printer, which will hopefully happen later this year.

Aside from that the only other job will be deploying the new sensors once they are built. This will mean running all the remaining power cables through the roof space, so lots of crawling around up there (yay! /s).

That’s it for now. If you liked this post and want to receive further updates, please consider subscribing to the mailing list (in the sidebar) or RSS feed. You can also follow me on Twitter. Bye for now!

Home Assistant Automation in Depth: Making my mornings a little easier

5 Comments

I was recently asked by someone at work if I had succeeded in completely automating my house yet. My reply was something along the lines of: “Not yet, but I have got a few awesome automations running”. I then proceeded to give an explanation of my morning Home Assistant automation, which I’m going to document here.

Basically, when I get up in the morning and unplug my phone, the following Home Assistant automation is triggered:

  1. Turn on the lights in the kitchen
  2. Turn on the kettle (which has been set ready to go the night before)
  3. Wait five minutes (giving me time to stumble sleepily into the kitchen and it also happens to be roughly how long the kettle takes to boil)
  4. Turn on the TV
  5. Play a TTS based briefing, which includes the days weather forecast and current indoor temperature
  6. Play a two minute news summary from the BBC
  7. Five minutes after the news finishes, another automation kicks in, turning off the TV

Needless to say, my colleague was suitably impressed by this.

How does the house know I’m getting up?

I mentioned above that one of the first things I do when I get up is unplug my phone from its charger. This is key to triggering this Home Assistant automation. On my phone I have a Tasker profile running which will send a HTTP POST request to Home Assistant when the power state of the phone changes. This is used to update the status of a binary sensor in HASS, which reflects the charging state of my phone. Pretty simple.

Of course this wouldn’t work if I didn’t unplug my phone when getting up, or didn’t charge it overnight. Luckily I do, so this approach works well for me. I did initially play around with using a Baysian sensor to attempt to infer when I was in bed. However, that didn’t work very well for me due to lack of data. Most of the data points I had to use in the inference were things that I wanted to turn on from the automation, like the lights and kettle, which was unhelpful. Having extra sensors here, like motion sensors, would help.

The only downside of this approach (at least for me) is that Tasker is not Open Source and there is no viable FOSS replacement. I’m actively looking for replacements to Tasker for this. I’ll post an update if I ever find something that fits all my requirements.

Home Assistant Automation for My Mornings

The YAML code for this Home Assistant automation can be found in my HASS config, but I’ll go through it here. The first part is pretty standard:

Here we see the trigger from the charging status of my phone. This is followed by a couple of conditions which restrict when the automation runs. If we’re not home for any reason it doesn’t make sense for the house to start turning things on and talking to itself, so that’s the first condition. The second is there just in case I need to charge my phone at other times of day. This only allows the automation to run between 7am and 10am. If my phone needs recharging (and is then unplugged again) before 10am then I’d probably be more worried about what is happening to my battery than the house starting talking.

The second part of the automation just turns things on and waits for me:

The first script call there is a simple script to set the colour temperature of the kitchen downlight bulbs depending on the time of day, bluer during the day, redder in the evening. I use this script wherever these lights are turned on so that they are always at the right colour temperature. They also switch over in the evening, even if they are already on, thanks to my sunset automations.

Automating the Kettle

The next call there is simply turning on the kettle. This is connected to a dumb 433Mhz socket (controlled by OpenMQTTGateway). The reason for the dumb socket, rather than something smarter, there is that it’s the only one that I’ve found that is rated for the actual power draw of the kettle. It’s heating water from a 10A supply, so it actually pulls 10A. At 230V this is 2300W. Before anyone asks, I’m not taking account of the power factor here since I’d expect the kettle to be an almost purely resistive load. This socket is rated to 2400W and I’ve measured with a power monitor to make sure it stays below this. Most smart sockets on the market seem to only be rated up to 2200W for some reason.

Of course the kettle has to be set up with water in it and turned on at the base for this to work. The only approach to this is to program myself to do this before I go to bed. With this in mind I have another automation which turns off the kettle and sends me a notification to remind me to set it up.

The final call here is the delay. I played around for a while with the value and 5 minutes seems to be about right. Once I get some more sensors installed (door and/or motion sensors), I’ll tweak the rest of the automation to be triggered on the first time someone enters the kitchen.

Making the house talk

Now we get to something more interesting:

Most of this references my TTS package, which manages TTS announcements in my home. TTS here is provided by PicoTTS for its offline capability. I don’t want anyone listening in to my highly private weather forecast announcements. I’ve been told the voice is a little robotic, but I’d rather my computer sounds like a computer than a real person. Right now all my TTS announcements are sent to the Kodi instance running on the living room TV for playback.

The first thing to do here is to enable TTS notifications which are turned off overnight via an input boolean. All my TTS notifications go through a script which checks this is enabled before talking. This also handles turning on the TV ready to speak:

Before we get to the actual morning briefing there is one more check to do. This handles the case that I get a lie in and the rest of the family are up before me (rare!). In this case it’s likely that the TV is already playing something, so we can check if it is idle before continuing.

The Morning TTS Briefing

Now comes the TTS briefing, which is implemented by the following script:

This gives me a pretty good overview of the day. At some stage I’d like to add notification of upcoming calendar events from Nextcloud via CalDav, but I haven’t got around to it yet. The variable greetings based on time of day add some complexity to the templating. These aren’t necessary for this application, but they enable easy reuse of the script for other purposes. Perhaps a returning home announcement?

The next part of the automation is a bit of a hack. The problem appears to be that HASS will continue executing the automation without waiting for the TTS to finish. For now I wait a fixed delay after each part of the announcement before starting the next. Waiting for the media player to change state back to idle would be better, but I haven’t worked out how to do that yet. I haven’t got the timings quite right since the announcement time varies slightly (depending on the weather!). There is a little bit of delay between each part, but it’s acceptable.

The next part does a simple TTS announcement which introduces the news, bypassing the main TTS script because we just ran it. In this case everything is ready and we don’t want the extra delay that script introduces.

Playing the news

The BBC have a nice two minute news summary, which is available as an MP3 file. This will play fine directly from the web via Kodi and Home Assistant’s media_player.play_media service, if you can get the URL, which is something the BBC don’t make particularly easy. There is no podcast RSS feed for this as there are for many other BBC shows. They also seem to be going out of their way to deliberately obfuscate the URL. My script has broken once already in the <6 months I’ve had it running. Since it’s probably subject to change in the future I won’t embed my URL extraction script here, just follow the link to the latest version. Suffice to say Python+BeautifulSoup to the rescue!

This script is run via a command line sensor in HASS, which updates once per hour. Again, this is to make the resulting script more re-usable – it will always play the latest news:

Then there is the corresponding HASS script, which sends the URL to the media player:

And that’s pretty much it.

Conclusion

This setup has been working pretty well for the last few months, with a couple of minor hicups caused by external factors. The change to the BBC news obfuscation was one. The other was an issue with Node-RED on the Pi connected to the TV, which caused it not to be able to turn on automatically).

The only other issue I have with this is with regards to the TV volume. The TV remote control does not pass through button presses of the volume keys via HDMI CEC. Therefore, we use the TV volume as the master, with the other volume controls in Kodi/Alsa set to maximum. The problem with this is if it is left on a high volume the night before you’ll get a very loud TTS announcement in the morning! Since the volume level on the TV is not controllable from HASS I can’t adjust the volume to an acceptable level before playing the TTS. The solution to this is to use the Kodi volume control as the master volume, which is controllable from HASS. This requires a new remote for use with Kodi and I’m still looking into options here.

Further Improvements

Aside from the improvements listed above, I’d also like to expand the information given in the TTS briefing. I’ve already touched on adding calendar events. Adding UV exposure information would also be very useful, given the lethal NZ sun.

Hopefully, someone will find some of the approaches and information in this post useful. There are a lot of example configurations available for HASS now. However, it’s often hard to piece the parts together that make everything work for a specific Home Assistant automation without a detailed explanation. This is something I’d like to do more of for my own setup, so hopefully there will be some more posts covering other parts of my configuration in future. Until then, bye.

How my Chromecast breaking was the best thing that ever happened to my TV

Leave a comment

As detailed in my recent self-hosting update, I’ve been using a Raspberry Pi running OSMC and Kodi as my frontend for TV recordings and for locally streamed media from Emby, since moving into the new house. We’ve supplemented this with a Chromecast to allow us to access Netflix and a local streaming service Lightbox. This has worked well and integrates with Home Assistant reasonably well, so I can automatically dim the lights, etc. when we are watching TV in the evening.

That was until the Chromecast started behaving oddly.

It started as occasionally corrupted audio when starting a new stream (basically the audio would sound like everyone had been breathing helium). Each time this occurred it was remedied by rebooting the Chromecast, at first by cycling the power, then as the problem persisted via a Python script wired to a button in Home Assistant. This went on for a month or so before things got worse.

The next problem was the Chromecast just flat out refusing to load anything from Lightbox. I spent an evening debugging this to have the thing fall off the network and refuse to come back. It must have automatically recovered itself because the next day it was back and working fine. Then a couple of days later it started to have similar issues again, only now with various HDMI picture issues (not detected or video stained pink).

Clearly it was on it’s way out (suspiciously it was just over a year old, which puts it just out of warranty). Having had enough I unplugged the thing and started to look for other options. Having paid $109 for it to last only a year, I wasn’t happy to buy another Chromecast (I had bought the Chromecast Ultra, but only because it was the only model with built in Ethernet).

Aside: The insidious Chromecast ecosystem

As someone who generally prefers FOSS options wherever possible and has no love of DRM, I’ve always had issues with the Chromecast. That said, as someone who wants attain the media I watch via legal means I appreciate that it allowed me to do that. I also liked the ability to control it from my phone as well as play/pause/stop streams via the TV remote and the aforementioned integration with Home Assistant. Basically, I saw it as a necessary evil.

What I didn’t appreciate is what it does to your phone. Before you have even set the thing up you have to install the proprietary Google Home app (why it can’t have a web interface for configuration I don’t know), then every streaming app that supports it is proprietary (even the Emby one), which left me with a gaggle of proprietary apps on my phone which is mostly otherwise populated with Open Source apps from F-Droid. This severely limited my ability to go GApps free, which has been something I’ve wanted to do for quite some time.

So if I could find another option that didn’t rely on my phone, I could get rid of all these horrible apps (some of which I even have to have Magisk installed in order to persuade them that I don’t have root access).

Meanwhile, back at the plot…

Faced with replacing the Chromecast I had two options. The first was to plug the not so smart TV back in to the network and use the built in apps. This was sub-optimal as it didn’t integrate with Home Assistant, couldn’t be controlled from my phone and the Lightbox app on that TV has broken at least once (in fact I don’t even know if it works now, since I went back to using the Chromecast instead).

The second option was to get Kodi to do it all (I guess there was really a third option which was to go out and find some other streaming device, but I really didn’t want to waste my money again).

Kodi To The Rescue!

To cut a long story short, I managed to get everything working with Kodi over the course of a Sunday afternoon. I already knew there was a Netflix addon (requiring Kodi 18), which I’d been meaning to try, but I didn’t know of a Lightbox addon. A quick search turned up Matt Huisman’s Lightbox Addon, which works great (I’d already used his TVNZ OnDemand and 3NOW addons).

Getting Netflix working turned out to be a bit of a pain, since I had to upgrade to the Kodi 18 Alpha release. I followed these instructions, which didn’t work to start with, but that turned out to be down to a corrupted SD card (weirdly the card was fine in normal use but didn’t like installing new packages). After grabbing a new card and restoring from a backup image I was able to update successfully and install the Netflix addon, which works flawlessly.

The Good

Overall, I’m really happy with this setup. The alpha version of Kodi is surprisingly stable (on par with the release version from my experience so far but YMMV), notwithstanding a couple of bugs which I’ll come to shortly. Netflix and Lightbox work pretty much flawlessly and I’m appreciating the newly unified and simplified media system. I’ve already started implementing further integrations with the home automation, which will be the subject of another post.

The Not So Good

I mentioned above that there are a couple of bugs, but I actually suspect that both issues I’m seeing are down to a common cause. The two issues I’m seeing are both related to TV recordings from TVheadend, with audio sync issues as well as raised RPi temperatures on HD recordings and dropped frames on SD recordings. I still need to get around to updating to the latest nightly release and gathering the debug logs required to submit a proper bug report, but since the TV is a ‘production’ system I haven’t got to this yet. This is the kind of issue, that whilst annoying, isn’t a show stopper and that I would fully expect to be fixed by the final release of Kodi 18.

The only other not so good point is that whilst the Netflix and Lightbox plugins are excellent, navigating through the menus is a little slow. I’m putting this down to the need to fetch the listings over the network every time and probably even scrape the respective websites. I would assume that neither site provides a proper API given how generally hostile streaming services are to third party integrations. Perhaps this could be mitigated either in Kodi or the addons by caching the data for a period of time, since it doesn’t change that often. This definitely isn’t a criticism of either of these addons, I’m impressed that they work as well as they do given their adverse environment. Luckily playback in both is flawless.

Conclusion

Again, I’m really happy with this setup. It’s finally given me an almost 100% Open Source (less the binary blobs required for Widevine DRM) media setup, which doesn’t compromise on functionality and sources all the content via legal means. Kodi gets an undeservedly bad reputation in the media for being a platform which enables piracy, something which the project developers have quite rightly distanced themselves from. Having more addons for legitimate services will help to give the platform a better name (of course if the media industry would wake up and just offer DRM free media at a reasonable cost [i.e. not the same price as a physical copy], that would be even better – but I don’t see that happening any time soon). The most annoying thing about this bad reputation for me is every how to guide for Kodi advertising VPNs (of varying levels of dodgyness) at you, as if the copyright police are going to bang down your door for using Kodi with your own media or a legitimate streaming service.

I’m finding this setup to actually be more feature full than the previous set up. This is obvious when you think about it, since with everything running through Kodi all my media benefits from the huge amount of work that has gone into that project over the years. Whereas, with the siloed approach taken by the individual streaming services they are all doomed to reproduce features that may be in competing services or have been features of established media players since the beginning of time. One really nice feature is that Kodi will make the full metadata of media playing via addons available via it’s various API’s which means that remote control apps can see it, but also that it gets pushed through to Home Assistant. This was hit and miss with the Chromecast (Lightbox would provide some metadata, Netflix would provide none).

Basically, this setup is what the smart TV was meant to be, before the interests that compete with producing the best technical solution got their hands on it.

I mentioned earlier that I’m already planning a follow up post to this one. The upcoming post will detail some of the integration work I’ve been doing to integrate my media setup with Home Assistant. Please stay tuned for that in the coming weeks. Until then, bye.

HDMI CEC Flow

HDMI CEC for Home Assistant with Node-RED

1 Comment

I set out on a Sunday morning thinking this was going to be a quick project and, not having decided on a blog topic for this week, it seemed like the ideal candidate. I was wrong – about it being a quick project, hopefully not about it being a reasonable subject for a blog post.

This post is brought to you by issue #12846 in Home Assistant (and the letter ‘C’). That is to say, one of my automations was broken by this issue, which has been sitting open on GitHub since the beginning of march with no progress. I don’t want this to sound like the usual “user of Open Source application complains about free stuff”, because I’m not actually complaining. I understand that software breaks and sometimes there aren’t the resources available to fix it. The solution to this is to get more developers paid to work on Free and Open Source Software (but that’s entirely a discussion for outside of this post).

Actually, this post is here to offer a solution (or at least a temporary one) to the issue, outside of Home Assistant, since I couldn’t fix it myself (I took a look at the code in question and I couldn’t work it out – it needs to be done by someone with more familiarity with the Home Assistant core).

My solution is to use Node-RED along with the HDMI CEC nodes to create an auto-discovered MQTT switch with which I can turn on and off my TV. So, let’s get into the flow…

The Flow

HDMI CEC Flow

The HDMI CEC Switch Flow

This flow runs on an instance of Node-RED running on my OSMC based Raspberry Pi sitting behind my TV (for those keeping up at home, this makes two NR instances on my network – so far). Currently, this is the only flow running on this instance, but I’m considering what else I can run now that I have Node-RED available there. I installed Node-RED on OSMC using the official install/upgrade script. I had fully expected installing Node-RED under OSMC to be a major pain, but it turned out to just amount to running that command.

After the install had finished, I created a user for Node-RED since I like it to run under it’s own user and updated the systemd unit file accordingly. I then installed the CEC nodes linked above from the palette manager. Here I ran into a minor bump in the road in that the CEC nodes couldn’t execute the cec-client program. As it turned out the location of that binary is in a weird location on OSMC, so I added the following in the systemd file to set this up:

I also needed to add my new Node-RED user to the video group to allow access to the CEC device:

Where I really got stuck was playing around with the example flow for the CEC nodes. It wasn’t that it didn’t work as advertised, it was that it broke the CEC command passthrough to Kodi running on the same machine, rendering my TV remote useless within Kodi. Many hours, much futile searching and playing with cec-client later, I still wasn’t any closer to a solution. I knew it must work, because somehow the pycec script I was using previously is able to send an receive CEC packets without interfering with Kodi.

The breakthrough was dropping both a CEC-In and a CEC-Out node into my flow and only grabbing a few CEC packet types in the filter of the input node. I say ‘breakthrough’ – this works most of the time, but it throws a few errors and warnings on start up. I found it to be most reliable when I immediately restarted Kodi after deploying it – this also helps Kodi to regain its CEC connection if necessary.

So How Does It Work?

Oh, yeah. I was going to talk about the flow, but I kinda got sidetracked there.

Well, it’s pretty simple there are two sequences in the flow. One which handles the switch state and MQTT discovery configuration (bottom) and one which handles the incoming commands over MQTT and sending the corresponding CEC commands.

Let’s start with the bottom one:

This sequence has two input paths, the bottom of these executes on start up (or at deploy time) and sends the Home Assistant MQTT Discovery configuration, using the same technique I used in my volcano sensors. The start up message also passes through a 3 second delay before passing to an exec node, which restarts Kodi. I added the following to my sudoers file (via visudo), to allow this:

The top input path receives incoming CEC messages of the type REPORT_POWER_STATUS. In my setup, this only receives power messages from the TV, but you may receive messages from other devices on the bus, in this case you can add a check on the source address of the packet in the following function node (clue: the TV is usually address 0).

The message passes through a function node, which converts the power status to the switch status expected by HASS and also sets the MQTT topic:

Both input paths are connected to a common MQTT output node to send their respective messages (config and state) out to Home Assistant.

The top sequence simply subscribes to the command topic from HASS and determines whether the command was on or off. The JSON payload for the CEC command is then set respectively in either branch – this JSON is taken directly from the example flow linked above. Then we pass this out to the CEC adapter – done. When the device acts upon the CEC command it should send its new power state back through and update the state of our switch. The state will also be updated if you turn on the TV by other means, e.g. the remote.

Pure JSON

This JSON was made in clean, green New Zealand from 100% natural ingredients (electrons):

Bonus: Home Assistant Automation Rules

Here are the Home Assistant automations that I’m using with this. Basically I’m turning off the TV five minutes after either Kodi or the Chromecast stops playing, unless it started again in the meantime:

This uses a timer, which is defined as:

Done. Now we can be lazy/forgetful about leaving TV on and also not waste power. Mission Accomplished.

Conclusion

Hopefully, someone will find time to fix the bug above. I’m probably going to stick with this regardless because I had some other issues running pyCEC on top of OSMC – mainly because they don’t build the libcec bindings for Python 3 by default. I had some custom patches to do this, but it would break (in one way or another on every update). Hopefully, this solution should be more robust. Also, the MQTT connection used in this solution runs over TLS (rather than the unencrypted TCP of the pyCEC network mode), so there is a little security win. Plus, as I already mentioned, now I have a Node-RED instance on a Pi in my living room.