Road to Docker Part 2

My Road To Docker – Part 2: My Home Automation Stack

This post may contain affiliate links. Please see the disclaimer for more information.

This post is part of a series on this project. Here is the series so far:


In my first post of this series, I outlined my plan to convert my infrastructure over to a layered setup. This would consist of virtual machines (in various VLANs), with most of the services running in Docker. This post details the second stage of my road to Docker, although really was is the first stage since I’m writing these out of order! I actually converted my home automation systems over to Docker before tackling the web stack.

The motivation behind upgrading the home automation system first was to do it at the same time as I did a large update to Home Assistant, since I’d been holding back on updating. The main reason for this was the switch to Lovelace as the default UI, which I was dreading. As it turned out, I waited long enough for the awesome HASS developers to make all my problems go away (or at least the Lovelace related ones).

System Summary

I’ve written about my home automation setup before, but here is a brief recap of what I’m running (only the server side stuff):

I had also been running InfluxDB and Grafana. However, something broke in my setup and I hadn’t got around to fixing it. I therefore decided to cut my losses with that and not reinstall it (for now).

Finding Docker Images

Luckily for me, the four main components of my system all have official/recommended Docker images available. This was useful as I’m always pretty reticent to use some questionably maintained image from the Docker Hub, mainly due to the lack of security updates. I also wanted to avoid building custom Docker images for now, until I work out a decent update strategy.

In addition to the four services above I wanted to run the ESPHome dashboard in order to manage my devices better. I had previously just been using the command line tool to build and upload to them. This also has an official Docker image.

Road to Docker Part 2
Looks like I have a few devices to update!

I also ended up running a MaryTTS container to replace PicoTTS that I had been running for my voice announcements. This is due to the lack of PicoTTS inside the HASS Docker image. It was recommended that I use the silversniper/marytts image. Looking at this image, it hasn’t been updated in three years which. This reinforces my point about random images from Docker Hub. Luckily, this isn’t an externally facing application so it isn’t too critical from a security standpoint. However, I think I’ll look into updating that at some point.

Stacking Containers, again…

I set up a new clean VM inside my home automation VLAN. I was a little reserved about doing this, since it means everything in that VLAN (most of which is blocked from the internet) can see the full HA server. However, my main worry over not doing that was the mDNS used by ESPHome. If I can get Avahi/mDNS working across VLANs at some point I will move it. I still have a big network re-organisation to do, so hopefully it will get done then.

The full docker-compose.yml file for my new stack is given below:

There’s nothing particularly earth shattering here. The main point of interest is that I mount the voice file for my preferred MaryTTS voice inside the container. Actually finding the voices is a little interesting. The official way to download them is a GUI tool that won’t run inside the container. I eventually found the XML file which lists all the available voices and extracted the URL of the one I wanted (the online demo helps to decide).

The only other parts worth noting are that I mount all my volumes under /mnt/docker-data, which is an NFS share onto the ZFS array of the virtual machine host. This then gets rolled into my normal backups. I also didn’t bother with a reverse proxy for any of this, since I already have one for HASS sitting in my DMZ (yet to be Dockerised). The other services just get accessed via the machine hostname and port since they are only used internally.

Sometimes Waiting Pays

I didn’t run into any issues particular to setting this up in Docker. At this point I think it’s a pretty well trodden path and this setup is pretty much standard. I did however run into several issues with upgrading to the latest Home Assistant.

First, lets tackle the elephant in the room – Lovelace. I was really worried going into this that it was going to be a huge amount of work. My mind was somewhat put at rest by seeing the UI editor in action via misperry’s video. When I actually came to it, the migration process automatically re-created my existing UI pretty much perfectly. Lesson learned: there is something to be said for waiting for mature software, rather than jumping on the new shiny thing immediately!

Lovelace itself is awesome! The ease of configuration has made me actually focus on making my HASS UI nicer rather than just the bare minimum I could get away with that I had previously. In the screenshot below, you can see my new “Outdoors” panel. This contains weather information, outdoor related sensor readings and a couple of local webcam views.

Road to Docker Part 2
I probably should have taken this screenshot during the day

Remaining Issues

Most of my remaining issues were due to the Home Assistant “Great Migration”. This resulted in a load of entity IDs changing in various components. Obviously, this resulted in my having to update my configuration to change all the names. It took a little while to troubleshoot. This was because if the changed name is used in an automation, the automation has to actually fire to cause an error. In many cases it also just won’t fire, if the name is used in the triggers for the automation.

The final major issue I encountered, was with my frankly awesome vacuuming robot, which appeared to stop reacting to service calls in HASS. The underlying issue appears to be the Botvac D3 returning a different error message than the D7 that the library was tested with. So far this hasn’t been fixed, but I’m currently using the category: 2 workaround suggested and that’s working fine. I think I’ll have a look into fixing that issue and submit a PR when I get time.

Managing Updates

Managing updates to Docker images has always been an bit of an issue for me. In the past I’ve used Watchtower with some success. However, due to the capacity for breaking changes I want to manage HASS updates more carefully. It was suggested to me to just use a bash script which I can run periodically to do this. This isn’t something that had occurred to me before, probably because it’s so simple! Here’s the script I’m using:

This works beautifully and allows me to easily keep up with release to HASS and the other components, once I’ve verified that it’s reasonably safe to update.

Conclusion and Next Steps

Overall I’m pretty happy with how this move has turned out. Once the initial teething issues were all worked out the system has been very stable. I’m appreciating the extra utility of the ESPHome dashboard, which makes it very convenient to update my devices. It’s also great to be back on the latest version of Home Assistant.

In terms of next steps, I would like to give InfluxDB and Grafana another try. My main issue here has always been building the dashboards. It seems to be pretty tricky to get something both good looking and useful in Grafana. I also haven’t seen any pre-built dashboards for use with data from Home Assistant. Perhaps this is because they are so peculiar to individual setups.

I also have an LXD container running ZoneMinder. I’d like to re-deploy this as a Docker container on the same VM. Previously, I’ve not had too much luck running ZoneMinder in Docker. I’ll to see if the situation has improved when I tackle this migration.

I’m not actively working on any further migrations of other services to Docker at the moment, so there will probably be a break in this series for now. However, given my current success I’ll definitely be continuing on with this migration. I just want to work on some other projects for a while!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

Top shelves

Self Hosting Update

Since my first post on my self hosting setup, things have changed quite a bit. I thought I’d take the time to write up a few of those changes, having recently got much more interested in how I can improve my setup further. This has been stimulated at least in part by seeing some awesome setups browsing /r/homelab. There will be photos of the new setup at the bottom of this post.

So What’s Changed?

Well the first thing was that I moved house. This was a protracted move, with 4 months spent living at my parents place before moving into our new home. Due to space and other constraints I didn’t want to run the servers when living with them. Instead, I settled for playing around with a couple of Raspberry Pis in the mean time. One of these was a new Pi 3 bought specifically for the purpose of becoming a Kodi box. It does this quite nicely thanks to OSMC. The other was a Pi 2 which just had a testing setup of Home Assistant on for me to play around with.

Since moving into the new house, I’ve been building my self hosting setup back up. I think I’ve now surpassed level as I was at previously. Since everything had been offline for 4 months, I decided to make a clean break of things. After a back up I formatted the system drive of the main server and installed Ubuntu Server. This was with a view to running my services in LXD containers. This was made possible by the aforementioned Pi 3 becoming the main TV frontend, along with a Chromecast for Netflix duties. That meant the server could go fully headless for the first time and be relocated to the garage, where it can be attached to a noisy UPS.

What am I Running?

Currently, I’m running several containers on the server. These include:

  • A Home Assistant/Mosquitto/Node-RED container
  • A music server container running Mopidy+Snapcast for (eventually) multi-room audio
  • A Tvheadend container to replace Mythtv (not that I was unhappy with it, I just thought I’d try something new)
  • An Emby container for serving other media to Kodi (in future I’d like to add a second RPi/Kodi instance)
  • A CheckMK container to replace the previous built from source Nagios server
  • A couple of others for early stage testing of new projects

New Firewall

In addition to separating the main server from the media frontend I also invested in a new firewall box before moving into the new house. This was primarily due to the new house having a fibre connection. The USB Ethernet device on the old netbook I was using therefore became a bottleneck on Internet speed. I picked up one of those dual Ethernet Haswell based mini-computers from AliExpress.

This was originally running pfSense natively on the hardware. However, in order to try and get a little more out of the new hardware I’ve since swapped this out for a Proxmox host which runs pfSense in a VM (more on this in a future post). This runs really nicely and I’ve noticed that the case doesn’t get anywhere near as hot as it did running pfSense natively.  Potentially this is confirmation bias on my part! The average air temperature has changed somewhat due to it getting towards winter.

I’m also running another VM on this system, which is hosting a testing install of Nextcloud. I haven’t transferred this to ‘production’ yet, mainly due to lack of time to get back to it. I’m pretty happy with it and will probably re-deploy it into an LXC container (Proxmox uses straight LXC not LXD) in order to reduce the memory footprint. I should have gone for more RAM in that box!. The main winner on the Proxmox install has been the ease with which I can do complex networking as required for the virtualised firewall and my VLAN setup. This is mainly due to the integration with OpenVSwitch, which I like a lot.

A Proper Switch

Having had the foresight to install Ethernet throughout our new home, I’ve needed to invest in a proper switch since we moved in. For a while I made do with piggy backing together my two wireless access points which provided 5 ports each. With this arrangement I was able to cover all the basics of my network. However, I wasn’t able to make every Ethernet jack in the house live and had no room for expansion.

I recently bought a TP-Link TL-SG1024DE 24 port switch. Whilst not the best switch in the world it is pretty good value for money and will serve my needs for the foreseeable future. Configuration of the VLANs is a little clunky, compared to the OpenWRT configuration interface I was using previously. However, everything works once it’s all configured. The great thing is I’ve been able to connect every port in the house as well as all my other gear and still have a ton of ports left over. The only feature I am missing in this switch is SNMP for monitoring, but I’m reasonably confident of being able to scrape the web interface at some point. Overall this switch is great for self hosting, where perhaps you don’t need those advanced features.

The Future

Based on my positive experience with Proxmox, I’ll probably migrate the main server to that at some point in the future. I’ve really enjoyed using LXD on Ubuntu, but Proxmox just seems better suited to my needs. The one feature I will miss from Ubuntu Server as a host is the kernel livepatching, which is really cool. The main thing holding me back from this at the moment is having to migrate all the existing LXD containers to LXC as there doesn’t seem to be a clean way to do this. This means the migration will have to wait until I can get all the services deployable via Ansible, which I’m working on.

Photos

As promised here are the photos of my self hosting setup. I’m using some standard garage shelving as a rack stand in. This works pretty well as I don’t have any rack mount gear except the new switch:

My Self-Hosted Life

This post may contain affiliate links. Please see the disclaimer for more information.

For those that know me, I’ve made no secret of the fact that I believe that you are better off doing something yourself than outsourcing the task to someone else, especially in areas that you are interested in or have some expertise. For me this has particular value in the case of my computing. As a result, I have taken the decision to self-host as much of my online services as possible, rather than relying on the cloud (since that’s just someone else’s computer). I’ve been working on this for years (actually the whole time this blog has been dark and before) and at this stage I’m mostly there: almost all of my digital life is provided by Open Source software, running under my control.

This post will detail what I’m using and how it all fits together. I’m not going to go into technical specifics since otherwise this post would be huge, perhaps I’ll focus on some of that in future posts (feel free to make requests in the comments). Also, please note that my setup is by no means finished and probably never will be, it’s an ongoing project and it has become pretty much my main hobby to install and maintain this stuff.

In the Cloud

I’m going to start right here, with this blog, since that was where the whole thing really started. This blog existed well before my undertaking to self-host. In the early days it lived on a shared hosting plan provided by Dreamhost. The site has always run WordPress, although I’ve toyed with the idea of moving to a static site over the years, I’ve just never quite managed it. In 2011 I moved the site to a shiny new VPS provided by Linode, where it has lived ever since. There is also a Piwik install for tracking website stats (which I’ve blogged about before).

The main motivation behind the VPS was to install and configure my own mail server setup, something which I ranted about shortly after. This setup has be serving myself and various family members well since then, with really very little maintenance on my part (almost everything is automated).

There have been various other uses for the VPS over time, many of which haven’t stuck. Probably the most successful has been an installation of TT-RSS, which started life on my home server and at some point moved to the VPS for convenience of access. I’ve also dabbled with various chat applications, mainly XMPP based, but they’ve never really been that useful due to the network effect of no-one else using them! At this stage email has become my primary form of communication.

You might say that this is a bit of a cop out, since this all runs on a virtual machine, which itself runs on someone else’s computer. I would agree, however it’s a nice middle ground between going all out with your own servers and running everything in the cloud. To me the reality that the VPS is in the cloud is obscured by the ability to control every detail of its running software. Its also pretty nice for services which I want to be reliable, since Linode almost never skips a beat.

At Home

So the VPS is one thing and is really used for critical services or stuff that needs to be accessible to the wider Internet (like this site), but the real magic happens on my home servers (yes, there is more than one). My main server (now on its second hardware iteration) started life as a MythTV system and still does a great job in this respect. Many other services have been added over time, such as an MQTT broker (mosquitto), git server (gitolite+gitweb), a calendar/contacts server (Radicale) and file synchronisation (Syncthing). At some point I also switched out the MythTV frontend and replaced it with XBMC (now Kodi).

In the last couple of years I’ve been moving further down the home automation route, rather than just sensing and logging via MQTT. I’ve finally settled on Home Assistant as my automation controller and UI, along with an instance of Node-RED to do some miscellaneous processing. This all runs on the main server, with a Raspberry Pi 2 in the garage functioning as what I like to call ‘the gateway’ (it has a couple of radios and some sensors connected and runs another instance of Node-RED to shuttle this data to MQTT). In addition I have my home CCTV set up using a couple of webcams and MotionEye. One of the cameras is located remotely and connected to another Raspberry Pi (this time an old model B) and streams back to the main server with mjpg-streamer.

I also run a pfsense based firewall to protect my network and provide remote VPN access. This runs on an old netbook with an extra USB ethernet adapter. The internal network is partitioned using VLANs to provide a separate firewalled subnet for the home automation gear, some of which is cheap Chinese stuff which needs to be forcibly prevented from talking to the cloud. The networking gear consists of two TP-Link routers, flashed with OpenWRT which provides nice VLAN support. These have been configured to just provide switching and wireless access points and delegate all the firewalling, DNS and DHCP stuff to the firewall.

Within the last year or so I’ve been working on streamlining the management of all of this. The principle focus of this has been monitoring all the services I’ve got running. For this I’ve settled on Nagios, which I run in a separate VM hosted on the main home server. Although complex to set up, I can’t talk highly enough of Nagios, it’s brilliant and it saves me so much time just by knowing what is going on on my network. Email notifications from Nagios of course go via my own mail server! I’ve also played around with collectd, InfluxDB and Grafana for performance graphing, although I’ve yet to deploy this to everything.

Conclusion and The Future

So that was a probably non-exhaustive list of my self-hosting activities. I’m sure I’ve probably forgotten many things and of course there are the huge amounts of supporting software that I haven’t mentioned. As I said, I’m now at the stage where this meets almost all my computing needs although there are a few areas where I want to improve.

The main thing is automating and persisting my configuration, since I’m still mostly doing things manually. For this I’ve settled on a combination of Ansible and Docker. I’ve played extensively with both but haven’t really made much progress with deploying them for much more than testing purposes.

I’m also constantly evaluating new software to fill gaps in my ecosystem. I’m currently looking at Rocket.Chat and Hubot to provide a chat based interface for remote administration, but don’t have a usable system yet. I’m also toying with the idea of a Gitlab server to replace the gitolite+gitweb system and to utilise the CI in my automation strategy, but I’ve heard it requires a bit in terms of resources (incidently gitlab.com is really the only 3rd party service I heavily use).

That I am able to do this at all is a testament to the power of Free and Open Source software and cheap commodity hardware. I find it pretty awesome to think that almost every interaction I have online utilises my own infrastructure and that it works tirelessly for me 24/7.

I’m only just getting started documenting my setup here, for instance this post hasn’t touched on any of the client applications I use on my phone and desktop machines. I’m also going to do some more technical posts on various aspects as time goes on, so please stay tuned (or even subscribe to the RSS feed or mailing list!).

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

Reviving this Blog

Its been almost four years since I updated this blog and in that time I’ve been busy with life and family. I’ve still been working on blog worthy stuff, but all my spare time has been taken up with actual projects, rather than writing about them. Most likely this has just been a matter of priorities. I could have made time to blog, but wasn’t interested enough to do so. I’ve always kept the site running and software updated and I’ve watched the daily hits go down from several hundred to single digits.

Recently I’ve been thinking that I would like to get back into it. Its taken me a little while to set aside the time, but this post is the start of a new and (hopefully) sustained run of writing. How long this continues will really depend on the response I get, if I see people reading and responding to what I’m producing then I will feel justified in setting aside time for it.

I’m making this a little bit of a fresh start and with that in mind I’ve done some work on the site. All the old content will remain in place, since it still gets a few hits. However, I’ve updated the theme and added the option for readers to subscribe by email, since this seems pretty popular nowadays (you can still subscribe by RSS and that’s never going away). Also, the site is now only accessible by HTTPS thanks to Let’s Encrypt.

Content-wise I’ve tried to get a head start and currently have two further posts written and ready for publishing. These will be posted later this week and I’ll try to keep up the momentum. In terms of topics I’ll be covering there will be lots of stuff about self-hosting your own cloud services, some embedded stuff and general software and Linux stuff. I have a list of posts I want to write and I’m open to suggestions in the comments.

Let’s see how this goes…