home automation

HDMI CEC Flow

HDMI CEC for Home Assistant with Node-RED

Leave a comment

I set out on a Sunday morning thinking this was going to be a quick project and, not having decided on a blog topic for this week, it seemed like the ideal candidate. I was wrong – about it being a quick project, hopefully not about it being a reasonable subject for a blog post.

This post is brought to you by issue #12846 in Home Assistant (and the letter ‘C’). That is to say, one of my automations was broken by this issue, which has been sitting open on GitHub since the beginning of march with no progress. I don’t want this to sound like the usual “user of Open Source application complains about free stuff”, because I’m not actually complaining. I understand that software breaks and sometimes there aren’t the resources available to fix it. The solution to this is to get more developers paid to work on Free and Open Source Software (but that’s entirely a discussion for outside of this post).

Actually, this post is here to offer a solution (or at least a temporary one) to the issue, outside of Home Assistant, since I couldn’t fix it myself (I took a look at the code in question and I couldn’t work it out – it needs to be done by someone with more familiarity with the Home Assistant core).

My solution is to use Node-RED along with the HDMI CEC nodes to create an auto-discovered MQTT switch with which I can turn on and off my TV. So, let’s get into the flow…

The Flow

HDMI CEC Flow

The HDMI CEC Switch Flow

This flow runs on an instance of Node-RED running on my OSMC based Raspberry Pi sitting behind my TV (for those keeping up at home, this makes two NR instances on my network – so far). Currently, this is the only flow running on this instance, but I’m considering what else I can run now that I have Node-RED available there. I installed Node-RED on OSMC using the official install/upgrade script. I had fully expected installing Node-RED under OSMC to be a major pain, but it turned out to just amount to running that command.

After the install had finished, I created a user for Node-RED since I like it to run under it’s own user and updated the systemd unit file accordingly. I then installed the CEC nodes linked above from the palette manager. Here I ran into a minor bump in the road in that the CEC nodes couldn’t execute the cec-client program. As it turned out the location of that binary is in a weird location on OSMC, so I added the following in the systemd file to set this up:

I also needed to add my new Node-RED user to the video group to allow access to the CEC device:

Where I really got stuck was playing around with the example flow for the CEC nodes. It wasn’t that it didn’t work as advertised, it was that it broke the CEC command passthrough to Kodi running on the same machine, rendering my TV remote useless within Kodi. Many hours, much futile searching and playing with cec-client later, I still wasn’t any closer to a solution. I knew it must work, because somehow the pycec script I was using previously is able to send an receive CEC packets without interfering with Kodi.

The breakthrough was dropping both a CEC-In and a CEC-Out node into my flow and only grabbing a few CEC packet types in the filter of the input node. I say ‘breakthrough’ – this works most of the time, but it throws a few errors and warnings on start up. I found it to be most reliable when I immediately restarted Kodi after deploying it – this also helps Kodi to regain its CEC connection if necessary.

So How Does It Work?

Oh, yeah. I was going to talk about the flow, but I kinda got sidetracked there.

Well, it’s pretty simple there are two sequences in the flow. One which handles the switch state and MQTT discovery configuration (bottom) and one which handles the incoming commands over MQTT and sending the corresponding CEC commands.

Let’s start with the bottom one:

This sequence has two input paths, the bottom of these executes on start up (or at deploy time) and sends the Home Assistant MQTT Discovery configuration, using the same technique I used in my volcano sensors. The start up message also passes through a 3 second delay before passing to an exec node, which restarts Kodi. I added the following to my sudoers file (via visudo), to allow this:

The top input path receives incoming CEC messages of the type REPORT_POWER_STATUS. In my setup, this only receives power messages from the TV, but you may receive messages from other devices on the bus, in this case you can add a check on the source address of the packet in the following function node (clue: the TV is usually address 0).

The message passes through a function node, which converts the power status to the switch status expected by HASS and also sets the MQTT topic:

Both input paths are connected to a common MQTT output node to send their respective messages (config and state) out to Home Assistant.

The top sequence simply subscribes to the command topic from HASS and determines whether the command was on or off. The JSON payload for the CEC command is then set respectively in either branch – this JSON is taken directly from the example flow linked above. Then we pass this out to the CEC adapter – done. When the device acts upon the CEC command it should send its new power state back through and update the state of our switch. The state will also be updated if you turn on the TV by other means, e.g. the remote.

Pure JSON

This JSON was made in clean, green New Zealand from 100% natural ingredients (electrons):

Bonus: Home Assistant Automation Rules

Here are the Home Assistant automations that I’m using with this. Basically I’m turning off the TV five minutes after either Kodi or the Chromecast stops playing, unless it started again in the meantime:

This uses a timer, which is defined as:

Done. Now we can be lazy/forgetful about leaving TV on and also not waste power. Mission Accomplished.

Conclusion

Hopefully, someone will find time to fix the bug above. I’m probably going to stick with this regardless because I had some other issues running pyCEC on top of OSMC – mainly because they don’t build the libcec bindings for Python 3 by default. I had some custom patches to do this, but it would break (in one way or another on every update). Hopefully, this solution should be more robust. Also, the MQTT connection used in this solution runs over TLS (rather than the unencrypted TCP of the pyCEC network mode), so there is a little security win. Plus, as I already mentioned, now I have a Node-RED instance on a Pi in my living room.

Mt. Taranaki

Home Assistant MQTT Discovery Sensors in Node-RED

1 Comment

Alternatively Titled: How I Made Home Assistant Aware of the Volcano Next Door

Mt. Taranaki

If this guy blows, we’re gonna have a bad day

As I’ve previously mentioned, I’m a big fan of Home Assistant’s MQTT Discovery feature. I’ve also historically been a fan of Node-RED and have recently been getting back into it, not least due to the uptick in interest in the platform in the HASS community. So, I decided to have a play around and come up with an implementation of an auto-discovered MQTT sensor in Node-RED and used it to pull some interesting data into Home Assistant.

Since moving to a different part of New Zealand last year I’ve wanted implement a sensor in HASS which would monitor the state of the local volcano. Luckily, GeoNet provide a nice API for getting volcanic alert levels for all the volcanic fields in NZ. I was initially going to write a custom component for doing this (and at some point contribute it back), but being generally even shorter on time than usual at the moment I never quite got there. That was until I was playing around with Node-RED and had a brain wave.

The Flow

I’m going to cut straight to the chase and show a screenshot of the flow I came up with and explain it below (the flow JSON can be found later in the post):

The full volcano data flow

The full volcano data flow

The start of the flow is pretty basic – a simple inject node which injects a timestamp (the payload is irrelevant) every 6 hours to kick off the flow. I didn’t want to hit the API endpoint too often since I’ve so far never seen the data change and if the mountain suddenly goes boom, I think I’ll have more pressing issues than whether my data is up to date.

Next, we have the HTTP Request node which goes out and performs a GET request to the URL given in the API documentation above. I enabled TLS support and opted to get the response data back as a parsed JSON object. Since the API returns data for all the volcanic fields in New Zealand, the next node just filters for the Taranaki/Egmont field that I am interested in, using the following code in a function node:

Basically this just iterates over all the features in the data and finds the one with the ID taranakiegmont and then substitutes it’s data in as the message payload. I also build the topic for the subsequent publish to MQTT based on the volcano ID.

The output of this function branches to another function node on one branch and a delay node on the other. The delay node here is used to make sure that the function node above runs and sends it’s output before the original message passes to the the MQTT publish node. The top function node is responsible for building the required configuration payloads and topics for the three sensors this will create in Home Assistant (one for each of the quantities in the data from the API). This is achieved with the following snippet of code:

This does the same thing for three new message objects, building a payload and topic for each. I use the ability of HASS to grab data from the payload of the main publish by specifying the state topic as the topic I built in the previous function node and a value template for each, pretty much exactly as in the HASS documentation. All three outputs of this node are passed to the MQTT publish node, which publishes with QoS 2 and the retain flag set. This means that whenever Home Assistant comes up after a restart it will see the values in both the configuration and state topics for these sensors and re-create them automatically. Attentive readers would have also noticed that I publish the configuration messages whenever I publish the state (every 6 hours). This doesn’t matter as HASS will just ignore the configuration messages for sensors which it has already discovered.

So, that’s it. With this in place the sensors should appear in Home Assistant:

Home Assistant Volcano Sensors

Note the reassuring zero for activity level!

The JSON:

As promised, here is the full JSON for the flow. To add this to your Node-RED instance copy it to your clipboard and go to Hamburger->Import->Clipboard in Node-RED and paste the JSON. You can select whether to import to the current flow or a new flow and then hit ‘import’ and you should see the nodes:

If you are importing this directly, you will need to configure your MQTT broker settings under the MQTT publish node before hitting ‘deploy’.

Wrap Up

That’s pretty much all there is to it. I hope this has demonstrated the concept of using Node-RED to create sensors in Home Assistant, without any changes to the HASS configuration. The flow presented is pretty simple but actually serves a useful purpose. Hopefully, you can come up with some uses of your own for this approach. Please feel free to share them in the comments below if you do, so that others may benefit from your ideas.

Thanks for reading. I’m working on a few more things with Node-RED so hopefully I’ll post about them soon. Bye!

Micropython Based Room Sensors: Part 1

Leave a comment

Having recently moved into a new house, I find myself with a profusion of Smart Lights (not everywhere yet, but in more locations than I have previously had). The main problem is that I currently lack a way to drive these intelligently. Right now I have a few automations that drive them based on sunset/sunrise times and media state through Home Assistant. This is mainly due to having almost no sensors deployed in the new house currently – something I’m aiming to address in this project.

I’m planning to build and deploy a set of ESP8266 based sensors across the house, with at least one in each room. The base hardware for this is the ESP8266 connected to a PIR motion sensors and DHT22 temperature and humidity sensor (so nothing ground breaking). This should give me temperature/humidity data for the whole house as well as motion events that can be used as a starting point to drive the light automation.

Why ESP8266?

I’ve gone through several options for the sensor platform on the way to settling on the ESP8266. I’ve tried out MySensors and even gone as far as building a battery powered prototype complete with PCB. Unfortunately I stuffed up the PCB design so that the radio didn’t work and never got around to working out the problem. Thinking further on this I find the software platform limiting. For example, any kind of OTA firmware update is difficult.

The main disadvantage of the ESP8266 is the power usage, but given that I now own this house and that these are permanent sensors I’m planning on powering them via a 12v supply run from the main server cabinet through the roof space to each sensor. There is more than enough power from a 12v 2A supply to provide power to all the sensors.

Also, it’s cheap and easy to work with!

Why Micropython?

I had originally looked at either writing some custom firmware using the Arduino ESP8266 core or using ESPEasy. I’ve tried out ESPEasy on my first prototype and it does what I need, but as an Embedded Software Engineer by day I felt I needed to be more adventurous.

I love Python as my go to language whenever the platforms don’t dictate something else (in the case of embedded this almost always ends up being C or C++). Recently I’ve been getting more engaged with the Python ecosystem via a couple of excellent podcasts (Talk Python and Python Bytes), so this was a good time to get back into writing some Python at home. Having played around with earlier versions of Micropython on both the ESP8266 and the ESP32 I had a good idea of its capabilities and knew it could do what I needed.

Current Status

As of right now, I have one prototype sensor running using a ‘bare’ ESP-12 module with adapter board, plus the sensors soldered on to some vero-board (photos below). Using the ‘bare’ ESP module added quite a bit of complexity since you are responsible for pulling the various lines required to program/run the chip up or down. It’s not difficult, but it’s more difficult than it should be. For that reason, after I’ve exhausted my one remaining ‘bare’ module/adapter I will be switching to Wemos D1 Mini modules.

In terms of the hardware, I encountered a couple of issues building the first prototype. Whilst the ESP8266 adapter board I used has a regulator to take 5V down to the 3V3 required for the ESP, I wanted to use 12V to reduce the voltage drop through the wires in the ceiling space. This meant I needed another regulator to go from 12V down to 5V. At first I used a spare linear regulator I had in my parts box for this. As it turns out this was a mistake, whilst it powered the ESP and sensors just fine, the whole thing got very hot (even with an extra heat sink). This also affected the reading from the temperature sensor. The solution was to order some cheap buck converter modules online, which did the trick without getting the slightest bit warm and are much more efficient.

The second issue is a weird one. Whilst re-programming the ESP from ESPEasy to Micropython, I noticed that after running the erase flash command the current going to the board would get very high (~400mA at 12V!), this would result in the ESP getting very hot. Seemingly this didn’t damage it because I did it several times and it always continued to work. In addition to this the Micropython firmware wouldn’t boot, instead just spewing incomprehensible (at any of the common baud rates) garbage to the UART. As it turns out, the chip was not being erased correctly, running an erase regions across the full extent of flash did the trick:

If anyone knows what the problem with these modules is, please get in touch in the comments. I couldn’t find any reference to this problem online and the (probably cloned) Wemos modules I have don’t have the same problem.

Anyway, here are some photos of the finished prototype:

Fully assembled prototype

Fully assembled prototype with PIR sensor connected.

Front of prototype

Front of prototype showing ESP8266 and adapter board

Back of prototype

Back of prototype, the wires in the bottom right corner are where it was re-worked after changing the power supply

The Software

The software is fairly basic, but has a few nice features. I use the boot.py script to load the configuration from the config.json file on the internal filesystem, then I connect to the wifi following the example given in the Micropython documentation. After this main.py is executed which connects to the broker and initialises the sensors. This is where things get a bit more interesting. I’ve written a minimal implementation of the Home Assistant MQTT Discovery spec, which so far supports binary sensors and standard sensors. This allows the sensors to come up in HASS without any configuration on the HASS server (except enabling MQTT discovery, which is a one time operation). Reading from the sensors themselves is fairly standard, using a pin change interrupt for the motion sensor and the standard dht module for the temperature/humidity. The use of uasyncio allows the DHT sensor to be polled at regular intervals. The code for the whole program is available via GitLab (also mirrored on GitHub). You can see a screenshot of the sensors in HASS below:

Home Assistant state card

Home Assistant state card showing the automatically added entities

In comparison to other embedded platforms/languages, writing a Micropython application is awesome. The power of Python makes you extremely productive. I wrote the whole software in a couple of hours, including the MQTT discovery implementation, using a Wemos board with no sensors connected as my development platform. When I uploaded this to the prototype board it hit a couple of exceptions in the sensor code, which previously hadn’t been exercised. The key thing here is that it hit exceptions! The resulting stack traces of course tell you where and why the exception occurred. What could have been a long and arduous debugging session turned into a few minutes of tweaking. The availability of uasyncio is also a really nice feature and allows direct knowledge transfer from CPython.

Future Work

There is still some work to do in order to finish the project, mostly on the hardware side. Obviously I need to make enough sensors to cover the house, which is just an ongoing assembly task. There is also the in ceiling wiring, assembly into cases and mounting. I’ll be sure to post an update on this as I make progress. There are also a couple of software improvements I would like to make. Firstly, I would like to add a meta sensor which makes use of the LWT feature of MQTT to report the status of the unit, but also still utilises HASS MQTT discovery. I’m also planning to spin out the current MQTT discovery implementation into it’s own Python module, which will be published to PyPI. The only other software task is to write a script to automate the deployment of the Micropython firmware, application and device specific configuration, which will be useful when programming multiple units.

Thats all for now. I’ll be posting further parts in this series as the project progresses, so subscribe to the feed, join the mailing list (above) or follow me on Twitter to keep up.

My Home Automation Setup

1 Comment

Welcome to this long awaited (by me at least) post about my Home Automation setup, specifically my Home Assistant configuration. It feels like I’ve been promising this post forever, so it’s great to finally deliver. Let’s get straight into it.

I’ve pushed my full Home Assistant configuration to GitLab and will be linking to various parts throughout this post. The config makes heavy use of secrets, to obscure sensitive details about my infrastructure (the sensitive files themselves are encrypted with git-crypt).

What is Home Assistant?

Home Assistant Logo

 

Unless you’ve had your head in the sand or just have no interest in home automation (why are you reading this?) you should know that Home Assistant (HASS) is an Open Source home automation server written in Python (3). What you might not know is that it has an awesome and very active set of developers, two weekly release cycle and a friendly and helpful community. At the time of writing it supports 668 different components. A component is the HASS term for an integration with an external device, tool or service. These components form the base for a device and manufactuer agnostic home automation system. Due to the sheer number of them, chances are you have several HASS compatible devices or service subscriptions, even if you are not into home automation.

I ended up trying out Home Assistant whilst looking for an alternative to OpenHAB. My reasons for switching away from OpenHAB were that I found the profusion of DSLs unweildy and the automation rules difficult to debug, so I never managed to get a satisfactorily working system. I haven’t followed the project since moving to HASS, so these issues might have improved. I would encourage anyone for whom HASS doesn’t appeal to head over and take a look.

After being initally sceptical that the YAML based configuration and automations in HASS would be flexible enough to do what I wanted, I was pleasently surprised by the level of functionality available at little effort. Things like Scenes, which had been tricky in OpenHAB, were easy in HASS! Overall Home Assistant just operates at a higher level of abstraction, which makes things simpler. Since I switched the project has gone from strength to strength and gets better with every release!

My Setup and Hardware

My Home Assistant Frontend

My Home Assistant Frontend

As previously documented Home Assistant runs in a VM on my main home server. It talks mostly to devices in an isolated subnet – so that my IoT devices are firewalled from both the main LAN and the internet at large. The firewall on this subnet blocks both incoming and outgoing traffic, preventing any of these devices from calling home without my say so. The only ports open are those required to allow the devices to speak to HASS or the MQTT server (plus DNS and DHCP to the firewall).

In terms of hardware, I currently have:

  • 3x Milight GU10 RGBW bulbs and the associated (crappy) wifi bridge (I actually have the older wifi bridge). These are located in the living room and provide the main overhead lighting as well as evening mood lighting when watching TV.
  • 2x Kankun SP3 wifi switches, for which I wrote the HASS platform integration. These are connected to some low power (500W) heaters which provide heating in the bedrooms.
  • 2x Raspberry Pis with various sensors connected (temperature/humidity and motion). These also have attached Webcams which stream to ZoneMinder via MJPG-Streamer. One of these also forms the RC-Switch gateway for the RF switches below.
  • 3x 433Mhz dumb RF switches connected to a couple of lamps and the kettle, these are actuated via an MQTT to RC-Switch bridge implemented in Node-RED running on the aforementioned RPi.
  • 2x home built sensors which publish sensor data (temperature, light and motion) from the bedrooms via MQTT.
  • 1x Cheap (and insecure) nondescript Chinese IP camera – yes, I did change the default password and this thing is well firewalled (incoming and outgoing). Again, this streams to ZoneMinder.

In addition, I have Home Assistant setup with my Kodi instance as a media player and use the DarkSky support for providing weather information. Presence detection is provided via OwnTracks and SNMP from my pfsense firewall.

For notifications, I’m using both SMTP and the native HTML5 notifications. I’m particularly pleased with the HTML5 notifications, which are a recent addition to my config. They are a bit of a pain to setup, but they finally spurred me on to getting a proper nginx reverse proxy set up for HASS and getting a proper TLS certificate with Let’s Encrypt. Once set up they a reliable, fast and look great on all our devices. I haven’t yet tried out the actions support, but it does open up some interesting possibilities for new automations.

One component which is conspiciously absent from my setup is any kind of physical control of the smart devices. Some devices (like the Kankun switches) will sync their state back to HASS when manually actuated, but most need to be directly addressed by HASS in order to prevent getting into an inconsistent state. Aside from implementing automations (detailed below), so that manual intervention isn’t required for common actions, we also use the HTTP Shortcuts app on our phones for reasonably quick switching of devices. The user experience is still not brilliant, so it would be good to have some physical switches for some things in the future.

Scripts and Automations

The ‘A’ in Home Automation is for automation. Up until now, I’ve described the infrastructure that provides me with (remote) Home Control. The real power of HASS is in its capability to act without direct user input, or to react to your behaviour (somewhat) intelligently. To this end I have several automations and scripts set up to perform various actions, based on what’s going on. In no particular order these are:

  • Execute an ‘Away Mode’ script when presence detection reports everyone has left. There is a corresponding ‘Deactivate Away Mode’ script for when someone returns home. The away mode script switches my cameras into motion detect mode (indoor cameras are also powered up, since they are powered down for privacy when we are home). It also sends a notification to tell us this has happened.
  • On weekday mornings turn on the lights if it is still dark outside and turn on the kettle (this will cause it to boil automatically if I remembered to set it up the night before). The lights get turned off a little while later. The kettle gets turned off at night, ready for the next day. This automation only executes if someone is home.
  • Switch the living room lights into Movie mode in the evening (after a certain time and if the sun has set), when pressing play in Kodi. If playback is paused or stopped the main lights come back up. I recently added transitions to these changes, which don’t work perfectly with the Milight bulbs but are better than the harsh change I had before.
  • Turn on the lights just before sunset, with an adjustment for cloud cover. Through a bit of experimentation I found that 20 mins before sunset was good for fine days and 40 mins was good for cloudy days. This automation only executes if someone is home.
  • Turn on the lights if someone comes home after sunset. This works really well with the combination presence detection via OwnTracks and SNMP – the lights are always on by the time we get in the house.
  • Send various notifications, mostly for motion detection if we are out or at certain times. Also sends me an email when HASS has an upgrade pending.
  • This isn’t really an automation, but I use the Generic Thermostat component to control the heaters in the bedrooms. This is filtered to switch off based on presence by a couple of template sensors which wrap the actual temperature sensors when someone is home and return 100°C when everyone is out, thus making sure the heaters stay off. I’m sure there is a better way to do this, but it works.
  • I also run AppDaemon, but currently only have a single app – OccuSim, which mimics our lighting patterns when we are away.

That’s pretty much it. Those automations cover most of our current use cases, but there are definately more to come and there will be much more as I add more devices to the setup.

The Future

There’s lots to add in future. In addition to solving the manual input problems, I’m very interested in replacing the unreliable RC-Switches with some Sonoff modules (running custom firmware) and adding some more for other appliances. I’m also interested in upgrading my Milight gateway to a homebuilt one based on the ESP8266/NRF24L01 modules. I also like the look off the Yeelight bulbs as a better alternative to Milight. In addition I’ll soon be moving my Kodi install to a Raspberry Pi which will allow me to play with some HDMI-CEC stuff. I also need to improve the design of my frontend pages in HASS.

I’ve looked into many of the available components within HASS for devices which may be applicable to my setup, including Z-Wave and the Zigbee based platforms (Hue, IKEA Tradfri, etc.). Many of these platforms either don’t meet my requirements in terms of security (most Z-Wave devices don’t appear to use encryption [not that Milight or RC-Switch are secure either, but at least they are only switching low power LEDs]), are too expensive (looking at you Hue) or have limited availability here in NZ (Z-Wave devices using the AU/NZ spectrum are limited in number and I can’t find anywhere shipping the Tradfri devices to NZ [for the record Tradfri fits my security requirements]). For these reasons, I’ve been working on more custom devices based on Arduino/MySensors (for battery powered devices) and ESP8266/Homie for mains powered devices. The key here will be building something reliable and getting the physical design right so it is robust and not unsightly. Some of the new Z-Wave Plus devices also look promising due to their encryption support (particularly the Aeotec devices e.g. this).

For all of the above, I’ll try to do write ups here when I can. It’s a pretty long list and should keep me going for quite some time!

I hope you’ve enjoyed reading this high level view of my home automation setup. I’ll happily receive suggestions for improvements in the comments, or field questions. You can follow me on Twitter if you are that way inclined or subscribe to the RSS or email updates (see sidebar). Thanks again for reading.

Monthly Update: February 2017

Leave a comment

Hello again. It’s been a busy (and short) month so I don’t have much to update on. Most of my work this month has gone into refactoring my Home Assistant configuration into something which is publicly sharable. This has mostly involved splitting the configuration into more logical chunks than the few monolithic files I had previously and extracting secrets out into a file protected by git-crypt. I’ve also been updating and improving aspects of my config as I go, particularly the automations. I’m not quite ready to share this since I still have a couple of things to clean up and also need to actually deploy and test the new configuration. Hopefully this will be posted on gitlab during March, with an accompanying blog post here.

I’ve also been working on another Home Assistant related task, which was to get AppDaemon working. This was specifically so I could run Occusim, which provides occupancy simulation (turns things on and off when you’re not there) for Home Assistant. It didn’t take me long to get this up and running, but my first live test of Occusim it didn’t work, due to me not removing the test command properly. Now that I’ve fixed that issue, it works great.

I think that’s pretty much it for now. Hopefully there will be more to share next month.