home automation

Making my mornings a little easier with Home Assistant

Leave a comment

I was recently asked by someone at work if I had succeeded in completely automating my house yet. I replied, something along the lines of: “Not yet, but I have got a few awesome automations running” and proceeded to give an explanation of my morning routine automation, which I’m going to document here.

Basically, when I get up in the morning and unplug my phone, the following automation is triggered on my Home Assistant instance:

  1. Turn on the lights in the kitchen
  2. Turn on the kettle (which has been set ready to go the night before)
  3. Wait five minutes (giving me time to stumble sleepily into the kitchen and it also happens to be roughly how long the kettle takes to boil)
  4. Turn on the TV
  5. Play a TTS based briefing, which includes the days weather forecast and current indoor temperature
  6. Play a two minute news summary from the BBC
  7. Five minutes after the news finishes, another automation kicks in, turning off the TV

Needless to say, my colleague was suitably impressed by this.

How does the house know I’m getting up?

I mentioned above that one of the first things I do when I get up is unplug my phone from its charger and this is key to triggering this automation. On my phone I have a Tasker profile running which will send a HTTP POST request to Home Assistant when the power state of the phone changes. This is used to update the status of a binary sensor in HASS, which reflects the charging state of my phone. Pretty simple.

Of course this wouldn’t work if I didn’t unplug my phone when getting up, or didn’t charge it overnight. Luckily I do, so this approach works well for me. I did initially play around with using a Baysian sensor to attempt to infer when I was in bed, but that didn’t work very well for me due to lack of data. The only data points I had to use in the inference, aside from my phone state were elements that I wanted to turn on from the automation, like the lights and kettle, which was unhelpful. Having extra sensors here, like motion sensors, would help.

The only downside of this approach (at least for me) is that Tasker is not Open Source and there is no viable FOSS replacement. I’m actively looking for replacements to Tasker for this so I’ll post an update if I ever find something that fits all my requirements.

The Automation

The YAML code for the automation can be found in my HASS config, but I’ll go through it here. The first part is pretty standard:

Here we see the trigger from the charging status of my phone, followed by a couple of conditions which restrict when the automation runs. Obviously, if we’re not home for any reason it doesn’t make sense for the house to start turning things on and talking to itself, so that’s the first condition. The second is there just in case I need to charge my phone at other times of day, basically only allow the automation to run between 7am and 10am. If my phone needs recharging (and is then unplugged again) before 10am then I’d probably be more worried about what is happening to my battery than the house starting talking.

The second part of the automation just turns things on and waits for me:

The first script call there is a simple script to set the colour temperature of the kitchen downlight bulbs depending on the time of day, bluer during the day, redder in the evening. I use this script wherever these lights are turned on so that they are always at the right colour temperature (and they switch over in the evening, even if they are already on, thanks to my sunset automations).

The next call there is simply turning on the kettle, which is connected to a dumb 433Mhz socket (controlled by OpenMQTTGateway). The reason for the dumb socket, rather than something smarter, there is that it’s the only one that I’ve found that is rated for the actual power draw of the kettle. It’s heating water from a 10A supply, so it actually pulls 10A, which at 230V is 2300W (before anyone asks, I’m not taking account of the power factor here since I’d expect the kettle to be an almost purely resistive load). This socket is rated to 2400W and I’ve measured with a power monitor to make sure it stays below this. Most smart sockets on the market seem to only be rated up to 2200W for some reason.

Of course the kettle has to be set up with water in it and turned on at the base for this to work. The only approach to this is to program myself to do this before I go to bed. With this in mind I have another automation which turns off the kettle and sends me a notification to remind me to set it up.

The final call here is the delay. I played around for a while with the value and 5 minutes seems to be about right. Once I get some more sensors installed (door and/or motion sensors), I’ll tweak the rest of the automation to be triggered on the first time someone enters the kitchen.

Making the house talk

Now we get to something more interesting:

Most of this references my TTS package, which manages TTS announcements in my home. TTS here is provided by PicoTTS for its offline capability because I don’t want anyone listening in to my highly private weather forecast announcements. I’ve been told the voice is a little robotic, but I’d rather my computer sounds like a computer than a real person. Right now all my TTS announcements are sent to the Kodi instance running on the living room TV for playback.

The first thing to do here is to enable TTS notifications which are turned off overnight via an input boolean. All my TTS notifications go through a script which checks this is enabled before talking (it also handles turning on the TV ready to speak):

Before we get to the actual morning briefing there is one more check to do, which handles the case that I get a lie in and the rest of the family are up before me (rare!). In this case it’s likely that the TV is already playing something, so we can check if it is idle before continuing.

Now comes the TTS briefing, which is implemented by the following script:

This gives me a pretty good overview of the day. At some stage I’d like to add notification of upcoming calendar events from Nextcloud via CalDav, but I haven’t got around to it yet. The variable greetings based on time of day add some complexity to the templating, which isn’t necessary for this application, but enables easy reuse of the script for other purposes (perhaps a returning home announcement?).

The next part of the automation is a bit of a hack. The problem appears to be that HASS will continue executing the automation without waiting for the TTS to finish. For now I wait a fixed delay after each part of the announcement before starting the next. Waiting for the media player to change state back to idle would be better, but I haven’t worked out how to do that yet. I haven’t got the timings quite right since the announcement time varies slightly (depending on the weather!), so there is a little bit of delay between each part, but it’s acceptable.

The next part does a simple TTS announcement which introduces the news, bypassing the main TTS script because we just ran it, so everything is ready and we don’t want the extra delay it introduces.

Playing the news

The BBC have a nice two minute news summary, which is available as an MP3 file. This will play fine directly from the web via Kodi and Home Assistant’s media_player.play_media service, if you can get the URL, which is something the BBC don’t make particularly easy. There is no podcast RSS feed for this as there are for many other BBC shows and they seem to be going out of their way to deliberately obfuscate the URL (especially since my script has broken once already in the <6 months I’ve had it running). Since it’s probably subject to change in the future I won’t embed my URL extraction script here, just follow the link to the latest version. Suffice to say Python+BeautifulSoup to the rescue!

This script is run via a command line sensor in HASS, which updates once per hour (again, this is to make the resulting script more re-usable – it will always play the latest news):

Then there is the corresponding HASS script, which sends the URL to the media player:

And that’s pretty much it.

Conclusion

This setup has been working pretty well for the last few months, with a couple of minor hicups caused by external factors (the change to the BBC news obfuscation being one and the other being an issue with Node-RED on the Pi connected to the TV, which caused it not to be able to turn on automatically).

The only other issue I have with this is with regards to the TV volume. Since the TV remote control does not pass through button presses of the volume keys via HDMI CEC we use the TV volume as the master (with the other volume controls in Kodi/Alsa set to maximum). The problem with this is if it is left on a high volume the night before you’ll get a very loud TTS announcement in the morning! Since the volume level on the TV is not controllable from HASS I can’t adjust the volume to an acceptable level before playing the TTS. The solution to this is to use the Kodi volume control as the master volume, which is controllable from HASS. This requires a new remote for use with Kodi and I’m still looking into options here.

Aside from the improvements listed above, I’d also like to expand the information given in the TTS briefing. I’ve already touched on adding calendar events, but adding UV exposure information would also be very useful, given the lethal NZ sun.

Hopefully, someone will find some of the approaches and information in this post useful. There are a lot of example configurations available for HASS now, but it’s often hard to piece the parts together that make everything work for a specific automation without a detailed explanation. This is something I’d like to do more of for my own setup, so hopefully there will be some more posts covering other parts of my configuration in future. Until then, bye.

How my Chromecast breaking was the best thing that ever happened to my TV

Leave a comment

As detailed in my recent self-hosting update, I’ve been using a Raspberry Pi running OSMC and Kodi as my frontend for TV recordings and for locally streamed media from Emby, since moving into the new house. We’ve supplemented this with a Chromecast to allow us to access Netflix and a local streaming service Lightbox. This has worked well and integrates with Home Assistant reasonably well, so I can automatically dim the lights, etc. when we are watching TV in the evening.

That was until the Chromecast started behaving oddly.

It started as occasionally corrupted audio when starting a new stream (basically the audio would sound like everyone had been breathing helium). Each time this occurred it was remedied by rebooting the Chromecast, at first by cycling the power, then as the problem persisted via a Python script wired to a button in Home Assistant. This went on for a month or so before things got worse.

The next problem was the Chromecast just flat out refusing to load anything from Lightbox. I spent an evening debugging this to have the thing fall off the network and refuse to come back. It must have automatically recovered itself because the next day it was back and working fine. Then a couple of days later it started to have similar issues again, only now with various HDMI picture issues (not detected or video stained pink).

Clearly it was on it’s way out (suspiciously it was just over a year old, which puts it just out of warranty). Having had enough I unplugged the thing and started to look for other options. Having paid $109 for it to last only a year, I wasn’t happy to buy another Chromecast (I had bought the Chromecast Ultra, but only because it was the only model with built in Ethernet).

Aside: The insidious Chromecast ecosystem

As someone who generally prefers FOSS options wherever possible and has no love of DRM, I’ve always had issues with the Chromecast. That said, as someone who wants attain the media I watch via legal means I appreciate that it allowed me to do that. I also liked the ability to control it from my phone as well as play/pause/stop streams via the TV remote and the aforementioned integration with Home Assistant. Basically, I saw it as a necessary evil.

What I didn’t appreciate is what it does to your phone. Before you have even set the thing up you have to install the proprietary Google Home app (why it can’t have a web interface for configuration I don’t know), then every streaming app that supports it is proprietary (even the Emby one), which left me with a gaggle of proprietary apps on my phone which is mostly otherwise populated with Open Source apps from F-Droid. This severely limited my ability to go GApps free, which has been something I’ve wanted to do for quite some time.

So if I could find another option that didn’t rely on my phone, I could get rid of all these horrible apps (some of which I even have to have Magisk installed in order to persuade them that I don’t have root access).

Meanwhile, back at the plot…

Faced with replacing the Chromecast I had two options. The first was to plug the not so smart TV back in to the network and use the built in apps. This was sub-optimal as it didn’t integrate with Home Assistant, couldn’t be controlled from my phone and the Lightbox app on that TV has broken at least once (in fact I don’t even know if it works now, since I went back to using the Chromecast instead).

The second option was to get Kodi to do it all (I guess there was really a third option which was to go out and find some other streaming device, but I really didn’t want to waste my money again).

Kodi To The Rescue!

To cut a long story short, I managed to get everything working with Kodi over the course of a Sunday afternoon. I already knew there was a Netflix addon (requiring Kodi 18), which I’d been meaning to try, but I didn’t know of a Lightbox addon. A quick search turned up Matt Huisman’s Lightbox Addon, which works great (I’d already used his TVNZ OnDemand and 3NOW addons).

Getting Netflix working turned out to be a bit of a pain, since I had to upgrade to the Kodi 18 Alpha release. I followed these instructions, which didn’t work to start with, but that turned out to be down to a corrupted SD card (weirdly the card was fine in normal use but didn’t like installing new packages). After grabbing a new card and restoring from a backup image I was able to update successfully and install the Netflix addon, which works flawlessly.

The Good

Overall, I’m really happy with this setup. The alpha version of Kodi is surprisingly stable (on par with the release version from my experience so far but YMMV), notwithstanding a couple of bugs which I’ll come to shortly. Netflix and Lightbox work pretty much flawlessly and I’m appreciating the newly unified and simplified media system. I’ve already started implementing further integrations with the home automation, which will be the subject of another post.

The Not So Good

I mentioned above that there are a couple of bugs, but I actually suspect that both issues I’m seeing are down to a common cause. The two issues I’m seeing are both related to TV recordings from TVheadend, with audio sync issues as well as raised RPi temperatures on HD recordings and dropped frames on SD recordings. I still need to get around to updating to the latest nightly release and gathering the debug logs required to submit a proper bug report, but since the TV is a ‘production’ system I haven’t got to this yet. This is the kind of issue, that whilst annoying, isn’t a show stopper and that I would fully expect to be fixed by the final release of Kodi 18.

The only other not so good point is that whilst the Netflix and Lightbox plugins are excellent, navigating through the menus is a little slow. I’m putting this down to the need to fetch the listings over the network every time and probably even scrape the respective websites. I would assume that neither site provides a proper API given how generally hostile streaming services are to third party integrations. Perhaps this could be mitigated either in Kodi or the addons by caching the data for a period of time, since it doesn’t change that often. This definitely isn’t a criticism of either of these addons, I’m impressed that they work as well as they do given their adverse environment. Luckily playback in both is flawless.

Conclusion

Again, I’m really happy with this setup. It’s finally given me an almost 100% Open Source (less the binary blobs required for Widevine DRM) media setup, which doesn’t compromise on functionality and sources all the content via legal means. Kodi gets an undeservedly bad reputation in the media for being a platform which enables piracy, something which the project developers have quite rightly distanced themselves from. Having more addons for legitimate services will help to give the platform a better name (of course if the media industry would wake up and just offer DRM free media at a reasonable cost [i.e. not the same price as a physical copy], that would be even better – but I don’t see that happening any time soon). The most annoying thing about this bad reputation for me is every how to guide for Kodi advertising VPNs (of varying levels of dodgyness) at you, as if the copyright police are going to bang down your door for using Kodi with your own media or a legitimate streaming service.

I’m finding this setup to actually be more feature full than the previous set up. This is obvious when you think about it, since with everything running through Kodi all my media benefits from the huge amount of work that has gone into that project over the years. Whereas, with the siloed approach taken by the individual streaming services they are all doomed to reproduce features that may be in competing services or have been features of established media players since the beginning of time. One really nice feature is that Kodi will make the full metadata of media playing via addons available via it’s various API’s which means that remote control apps can see it, but also that it gets pushed through to Home Assistant. This was hit and miss with the Chromecast (Lightbox would provide some metadata, Netflix would provide none).

Basically, this setup is what the smart TV was meant to be, before the interests that compete with producing the best technical solution got their hands on it.

I mentioned earlier that I’m already planning a follow up post to this one. The upcoming post will detail some of the integration work I’ve been doing to integrate my media setup with Home Assistant. Please stay tuned for that in the coming weeks. Until then, bye.

HDMI CEC Flow

HDMI CEC for Home Assistant with Node-RED

Leave a comment

I set out on a Sunday morning thinking this was going to be a quick project and, not having decided on a blog topic for this week, it seemed like the ideal candidate. I was wrong – about it being a quick project, hopefully not about it being a reasonable subject for a blog post.

This post is brought to you by issue #12846 in Home Assistant (and the letter ‘C’). That is to say, one of my automations was broken by this issue, which has been sitting open on GitHub since the beginning of march with no progress. I don’t want this to sound like the usual “user of Open Source application complains about free stuff”, because I’m not actually complaining. I understand that software breaks and sometimes there aren’t the resources available to fix it. The solution to this is to get more developers paid to work on Free and Open Source Software (but that’s entirely a discussion for outside of this post).

Actually, this post is here to offer a solution (or at least a temporary one) to the issue, outside of Home Assistant, since I couldn’t fix it myself (I took a look at the code in question and I couldn’t work it out – it needs to be done by someone with more familiarity with the Home Assistant core).

My solution is to use Node-RED along with the HDMI CEC nodes to create an auto-discovered MQTT switch with which I can turn on and off my TV. So, let’s get into the flow…

The Flow

HDMI CEC Flow

The HDMI CEC Switch Flow

This flow runs on an instance of Node-RED running on my OSMC based Raspberry Pi sitting behind my TV (for those keeping up at home, this makes two NR instances on my network – so far). Currently, this is the only flow running on this instance, but I’m considering what else I can run now that I have Node-RED available there. I installed Node-RED on OSMC using the official install/upgrade script. I had fully expected installing Node-RED under OSMC to be a major pain, but it turned out to just amount to running that command.

After the install had finished, I created a user for Node-RED since I like it to run under it’s own user and updated the systemd unit file accordingly. I then installed the CEC nodes linked above from the palette manager. Here I ran into a minor bump in the road in that the CEC nodes couldn’t execute the cec-client program. As it turned out the location of that binary is in a weird location on OSMC, so I added the following in the systemd file to set this up:

I also needed to add my new Node-RED user to the video group to allow access to the CEC device:

Where I really got stuck was playing around with the example flow for the CEC nodes. It wasn’t that it didn’t work as advertised, it was that it broke the CEC command passthrough to Kodi running on the same machine, rendering my TV remote useless within Kodi. Many hours, much futile searching and playing with cec-client later, I still wasn’t any closer to a solution. I knew it must work, because somehow the pycec script I was using previously is able to send an receive CEC packets without interfering with Kodi.

The breakthrough was dropping both a CEC-In and a CEC-Out node into my flow and only grabbing a few CEC packet types in the filter of the input node. I say ‘breakthrough’ – this works most of the time, but it throws a few errors and warnings on start up. I found it to be most reliable when I immediately restarted Kodi after deploying it – this also helps Kodi to regain its CEC connection if necessary.

So How Does It Work?

Oh, yeah. I was going to talk about the flow, but I kinda got sidetracked there.

Well, it’s pretty simple there are two sequences in the flow. One which handles the switch state and MQTT discovery configuration (bottom) and one which handles the incoming commands over MQTT and sending the corresponding CEC commands.

Let’s start with the bottom one:

This sequence has two input paths, the bottom of these executes on start up (or at deploy time) and sends the Home Assistant MQTT Discovery configuration, using the same technique I used in my volcano sensors. The start up message also passes through a 3 second delay before passing to an exec node, which restarts Kodi. I added the following to my sudoers file (via visudo), to allow this:

The top input path receives incoming CEC messages of the type REPORT_POWER_STATUS. In my setup, this only receives power messages from the TV, but you may receive messages from other devices on the bus, in this case you can add a check on the source address of the packet in the following function node (clue: the TV is usually address 0).

The message passes through a function node, which converts the power status to the switch status expected by HASS and also sets the MQTT topic:

Both input paths are connected to a common MQTT output node to send their respective messages (config and state) out to Home Assistant.

The top sequence simply subscribes to the command topic from HASS and determines whether the command was on or off. The JSON payload for the CEC command is then set respectively in either branch – this JSON is taken directly from the example flow linked above. Then we pass this out to the CEC adapter – done. When the device acts upon the CEC command it should send its new power state back through and update the state of our switch. The state will also be updated if you turn on the TV by other means, e.g. the remote.

Pure JSON

This JSON was made in clean, green New Zealand from 100% natural ingredients (electrons):

Bonus: Home Assistant Automation Rules

Here are the Home Assistant automations that I’m using with this. Basically I’m turning off the TV five minutes after either Kodi or the Chromecast stops playing, unless it started again in the meantime:

This uses a timer, which is defined as:

Done. Now we can be lazy/forgetful about leaving TV on and also not waste power. Mission Accomplished.

Conclusion

Hopefully, someone will find time to fix the bug above. I’m probably going to stick with this regardless because I had some other issues running pyCEC on top of OSMC – mainly because they don’t build the libcec bindings for Python 3 by default. I had some custom patches to do this, but it would break (in one way or another on every update). Hopefully, this solution should be more robust. Also, the MQTT connection used in this solution runs over TLS (rather than the unencrypted TCP of the pyCEC network mode), so there is a little security win. Plus, as I already mentioned, now I have a Node-RED instance on a Pi in my living room.

Mt. Taranaki

Home Assistant MQTT Discovery Sensors in Node-RED

3 Comments

Alternatively Titled: How I Made Home Assistant Aware of the Volcano Next Door

Mt. Taranaki

If this guy blows, we’re gonna have a bad day

As I’ve previously mentioned, I’m a big fan of Home Assistant’s MQTT Discovery feature. I’ve also historically been a fan of Node-RED and have recently been getting back into it, not least due to the uptick in interest in the platform in the HASS community. So, I decided to have a play around and come up with an implementation of an auto-discovered MQTT sensor in Node-RED and used it to pull some interesting data into Home Assistant.

Since moving to a different part of New Zealand last year I’ve wanted implement a sensor in HASS which would monitor the state of the local volcano. Luckily, GeoNet provide a nice API for getting volcanic alert levels for all the volcanic fields in NZ. I was initially going to write a custom component for doing this (and at some point contribute it back), but being generally even shorter on time than usual at the moment I never quite got there. That was until I was playing around with Node-RED and had a brain wave.

The Flow

I’m going to cut straight to the chase and show a screenshot of the flow I came up with and explain it below (the flow JSON can be found later in the post):

The full volcano data flow

The full volcano data flow

The start of the flow is pretty basic – a simple inject node which injects a timestamp (the payload is irrelevant) every 6 hours to kick off the flow. I didn’t want to hit the API endpoint too often since I’ve so far never seen the data change and if the mountain suddenly goes boom, I think I’ll have more pressing issues than whether my data is up to date.

Next, we have the HTTP Request node which goes out and performs a GET request to the URL given in the API documentation above. I enabled TLS support and opted to get the response data back as a parsed JSON object. Since the API returns data for all the volcanic fields in New Zealand, the next node just filters for the Taranaki/Egmont field that I am interested in, using the following code in a function node:

Basically this just iterates over all the features in the data and finds the one with the ID taranakiegmont and then substitutes it’s data in as the message payload. I also build the topic for the subsequent publish to MQTT based on the volcano ID.

The output of this function branches to another function node on one branch and a delay node on the other. The delay node here is used to make sure that the function node above runs and sends it’s output before the original message passes to the the MQTT publish node. The top function node is responsible for building the required configuration payloads and topics for the three sensors this will create in Home Assistant (one for each of the quantities in the data from the API). This is achieved with the following snippet of code:

This does the same thing for three new message objects, building a payload and topic for each. I use the ability of HASS to grab data from the payload of the main publish by specifying the state topic as the topic I built in the previous function node and a value template for each, pretty much exactly as in the HASS documentation. All three outputs of this node are passed to the MQTT publish node, which publishes with QoS 2 and the retain flag set. This means that whenever Home Assistant comes up after a restart it will see the values in both the configuration and state topics for these sensors and re-create them automatically. Attentive readers would have also noticed that I publish the configuration messages whenever I publish the state (every 6 hours). This doesn’t matter as HASS will just ignore the configuration messages for sensors which it has already discovered.

So, that’s it. With this in place the sensors should appear in Home Assistant:

Home Assistant Volcano Sensors

Note the reassuring zero for activity level!

The JSON:

As promised, here is the full JSON for the flow. To add this to your Node-RED instance copy it to your clipboard and go to Hamburger->Import->Clipboard in Node-RED and paste the JSON. You can select whether to import to the current flow or a new flow and then hit ‘import’ and you should see the nodes:

If you are importing this directly, you will need to configure your MQTT broker settings under the MQTT publish node before hitting ‘deploy’.

Wrap Up

That’s pretty much all there is to it. I hope this has demonstrated the concept of using Node-RED to create sensors in Home Assistant, without any changes to the HASS configuration. The flow presented is pretty simple but actually serves a useful purpose. Hopefully, you can come up with some uses of your own for this approach. Please feel free to share them in the comments below if you do, so that others may benefit from your ideas.

Thanks for reading. I’m working on a few more things with Node-RED so hopefully I’ll post about them soon. Bye!

Micropython Based Room Sensors: Part 1

Leave a comment

Having recently moved into a new house, I find myself with a profusion of Smart Lights (not everywhere yet, but in more locations than I have previously had). The main problem is that I currently lack a way to drive these intelligently. Right now I have a few automations that drive them based on sunset/sunrise times and media state through Home Assistant. This is mainly due to having almost no sensors deployed in the new house currently – something I’m aiming to address in this project.

I’m planning to build and deploy a set of ESP8266 based sensors across the house, with at least one in each room. The base hardware for this is the ESP8266 connected to a PIR motion sensors and DHT22 temperature and humidity sensor (so nothing ground breaking). This should give me temperature/humidity data for the whole house as well as motion events that can be used as a starting point to drive the light automation.

Why ESP8266?

I’ve gone through several options for the sensor platform on the way to settling on the ESP8266. I’ve tried out MySensors and even gone as far as building a battery powered prototype complete with PCB. Unfortunately I stuffed up the PCB design so that the radio didn’t work and never got around to working out the problem. Thinking further on this I find the software platform limiting. For example, any kind of OTA firmware update is difficult.

The main disadvantage of the ESP8266 is the power usage, but given that I now own this house and that these are permanent sensors I’m planning on powering them via a 12v supply run from the main server cabinet through the roof space to each sensor. There is more than enough power from a 12v 2A supply to provide power to all the sensors.

Also, it’s cheap and easy to work with!

Why Micropython?

I had originally looked at either writing some custom firmware using the Arduino ESP8266 core or using ESPEasy. I’ve tried out ESPEasy on my first prototype and it does what I need, but as an Embedded Software Engineer by day I felt I needed to be more adventurous.

I love Python as my go to language whenever the platforms don’t dictate something else (in the case of embedded this almost always ends up being C or C++). Recently I’ve been getting more engaged with the Python ecosystem via a couple of excellent podcasts (Talk Python and Python Bytes), so this was a good time to get back into writing some Python at home. Having played around with earlier versions of Micropython on both the ESP8266 and the ESP32 I had a good idea of its capabilities and knew it could do what I needed.

Current Status

As of right now, I have one prototype sensor running using a ‘bare’ ESP-12 module with adapter board, plus the sensors soldered on to some vero-board (photos below). Using the ‘bare’ ESP module added quite a bit of complexity since you are responsible for pulling the various lines required to program/run the chip up or down. It’s not difficult, but it’s more difficult than it should be. For that reason, after I’ve exhausted my one remaining ‘bare’ module/adapter I will be switching to Wemos D1 Mini modules.

In terms of the hardware, I encountered a couple of issues building the first prototype. Whilst the ESP8266 adapter board I used has a regulator to take 5V down to the 3V3 required for the ESP, I wanted to use 12V to reduce the voltage drop through the wires in the ceiling space. This meant I needed another regulator to go from 12V down to 5V. At first I used a spare linear regulator I had in my parts box for this. As it turns out this was a mistake, whilst it powered the ESP and sensors just fine, the whole thing got very hot (even with an extra heat sink). This also affected the reading from the temperature sensor. The solution was to order some cheap buck converter modules online, which did the trick without getting the slightest bit warm and are much more efficient.

The second issue is a weird one. Whilst re-programming the ESP from ESPEasy to Micropython, I noticed that after running the erase flash command the current going to the board would get very high (~400mA at 12V!), this would result in the ESP getting very hot. Seemingly this didn’t damage it because I did it several times and it always continued to work. In addition to this the Micropython firmware wouldn’t boot, instead just spewing incomprehensible (at any of the common baud rates) garbage to the UART. As it turns out, the chip was not being erased correctly, running an erase regions across the full extent of flash did the trick:

If anyone knows what the problem with these modules is, please get in touch in the comments. I couldn’t find any reference to this problem online and the (probably cloned) Wemos modules I have don’t have the same problem.

Anyway, here are some photos of the finished prototype:

Fully assembled prototype

Fully assembled prototype with PIR sensor connected.

Front of prototype

Front of prototype showing ESP8266 and adapter board

Back of prototype

Back of prototype, the wires in the bottom right corner are where it was re-worked after changing the power supply

The Software

The software is fairly basic, but has a few nice features. I use the boot.py script to load the configuration from the config.json file on the internal filesystem, then I connect to the wifi following the example given in the Micropython documentation. After this main.py is executed which connects to the broker and initialises the sensors. This is where things get a bit more interesting. I’ve written a minimal implementation of the Home Assistant MQTT Discovery spec, which so far supports binary sensors and standard sensors. This allows the sensors to come up in HASS without any configuration on the HASS server (except enabling MQTT discovery, which is a one time operation). Reading from the sensors themselves is fairly standard, using a pin change interrupt for the motion sensor and the standard dht module for the temperature/humidity. The use of uasyncio allows the DHT sensor to be polled at regular intervals. The code for the whole program is available via GitLab (also mirrored on GitHub). You can see a screenshot of the sensors in HASS below:

Home Assistant state card

Home Assistant state card showing the automatically added entities

In comparison to other embedded platforms/languages, writing a Micropython application is awesome. The power of Python makes you extremely productive. I wrote the whole software in a couple of hours, including the MQTT discovery implementation, using a Wemos board with no sensors connected as my development platform. When I uploaded this to the prototype board it hit a couple of exceptions in the sensor code, which previously hadn’t been exercised. The key thing here is that it hit exceptions! The resulting stack traces of course tell you where and why the exception occurred. What could have been a long and arduous debugging session turned into a few minutes of tweaking. The availability of uasyncio is also a really nice feature and allows direct knowledge transfer from CPython.

Future Work

There is still some work to do in order to finish the project, mostly on the hardware side. Obviously I need to make enough sensors to cover the house, which is just an ongoing assembly task. There is also the in ceiling wiring, assembly into cases and mounting. I’ll be sure to post an update on this as I make progress. There are also a couple of software improvements I would like to make. Firstly, I would like to add a meta sensor which makes use of the LWT feature of MQTT to report the status of the unit, but also still utilises HASS MQTT discovery. I’m also planning to spin out the current MQTT discovery implementation into it’s own Python module, which will be published to PyPI. The only other software task is to write a script to automate the deployment of the Micropython firmware, application and device specific configuration, which will be useful when programming multiple units.

Thats all for now. I’ll be posting further parts in this series as the project progresses, so subscribe to the feed, join the mailing list (above) or follow me on Twitter to keep up.