docker common containers

Quick Project: Splitting Docker Compose Projects

This post may contain affiliate links. Please see the disclaimer for more information.

Way back in the when I first started using Docker in earnest, I wrote about my web hosting stack. Recently, this has undergone an upgrade as I’m working on a new website which will be served from the same server. I took the opportunity to split the system up into multiple docker-compose projects, which makes deployment of further sites much easier. It allows me to manage the common containers from one docker-compose project and then each of the sites from their own project. This will be of further use in future as I move towards deploying these with Ansible.

The Approach

My basic approach here is to move my two common containers (my Traefik container and SMTP forwarder) into their own project. This project will create a couple of networks for interfacing to the containers from other projects. To create these networks I add the following to my common project docker-compose.yml:

networks:
  gateway:
    name: 'gateway'
  smtp:
    name: 'smtp'

Here I create two networks as per normal. The key is to give them a proper name, rather than the auto generated one that would be assigned by Docker. This will enable us to address them easily from our other projects. We then assign these to our common containers:

services:
  traefik:
    image: traefik:2.1
    command:
      ...
    volumes:
      ...
    ports:
      - "80:80"
      - "443:443"
      - "127.0.0.1:8080:8080"
    networks:
      gateway:
        aliases:
          # add hostnames you might want to refer to this container by
          - example.com
    restart: always

  postfix:
    image: boky/postfix
    ports:
      ...
    environment:
      ...
    volumes:
      ...
    networks:
      smtp:
        aliases:
          - postfix
    restart: always

Here I simply assign the relevant network to each container. The aliases section allows other containers on these networks to find our common containers by whatever name we specify. In the case of the postfix container this is to connect via SMTP. For the traefik container, adding hostnames which internal apps my need to refer to can help (for example with the WordPress loopback test).

External Projects

With this in place, the other applications can be moved out into their own projects. For each one we need to access the gateway and smtp networks in order to have access to our common services. These are accessed as external networks via the docker-compose.yaml file for our project:

networks:
  gateway:
    external: true
  smtp:
    external: true

We then go ahead and add our services to access these networks:

services:
  varnish:
    image: wodby/varnish:latest
    depends_on:
      - wordpress
    environment:
      ...
    labels:
      - 'traefik.enable=true'
      - "traefik.docker.network=gateway"
      ...
    networks:
      - gateway
      - cache
    restart: always

  wordpress:
    image: wordpress:latest
    depends_on:
      - mariadb
    environment:
      ...
    volumes:
      ...
    networks:
      smtp:
      cache:
      database:
    restart: always

Here I add my varnish cache, as per my previous article. The key thing here is to specify the label traefik.docker.network=gateway to allow Traefik to reliably discover the container. We then also make sure the container is added to the gateway network. I’ve also added a WordPress container, which is on the smtp network. This will allow sending of email from WordPress via the SMTP forwarder.

Conclusion

This is a pretty simple approach for better management of my increasingly complex web stack. As I mentioned earlier that the next step will be to deploy these projects via Ansible. In this case the common containers will become part of a role which can be used across my infrastructure.

The splitting out of the apps into their own projects has enabled me to duplicate my current WordPress+Varnish+Mariadb setup for the new site I’m working on. There will be more info to come about that site as soon as I am ready to share!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

mysensors node red proxy flow

Quick Project: MySensors MQTT Proxy with Node-RED

This post may contain affiliate links. Please see the disclaimer for more information.

In my previous exploration of MySensors, I encountered issues with the range of the nrf24l01+ radio. One of the simplest solutions to this problem is to move the gateway closer to the intended deployment of the sensors. In my case the sensor is positioned on the back wall of our house, just outside the master bedroom. Connected to the TV inside the master bedroom is a Raspberry Pi running Libreelec. I reasoned that I should be able to use this Pi to proxy the MySensors serial data to Home Assistant via MQTT, using the MySensors Node-RED nodes. With this approach I can make a new MQTT gateway using hardware I already have deployed.

It was also suggested that adding a 10-100uF 6.3/10V capacitor across the power line to the radio would also help with the range issues. I’m definitely going to do this, but the components haven’t arrived yet!

Hardware Setup

If you followed my previous post all you need to do here is plug in the serial gateway to the Pi! Otherwise you’re going to need to build a gateway (and maybe some sensors). Go check out my original post for more details.

It’s probably worth going through how this is going to work. Basically, the idea is that the RF data will be received by our existing MySensors gateway and sent via serial to the RPi. This will be read by Node-RED, re-encoded and sent via MQTT to Home Assistant which will still act as the MySensors controller:

Serial GW ⇒ Serial Connection ⇒ Node-RED (on RPi) ⇒ MQTT ⇒ HASS (Controller)

In the return direction this dataflow operates in reverse:

HASS ⇒ MQTT ⇒ Node-RED ⇒ Serial Connection ⇒ Serial GW

Deploying Node-RED on Libreelec

Libreelec is a cut down appliance-like Linux distribution, specifically for running Kodi. At first glance it seems like installing a third party program such as Node-RED would be difficult. However, Libreelec has Docker support via an add-on, which makes things as easy as running the command:

$ docker run -d --restart=always --device=/dev/ttyUSB0:/dev/ttyUSB0 --group-add dialout -v /storage/node-red-data:/data -p 1880:1880 --name node-red nodered/node-red:latest

This starts the Node-RED Docker container with access to the serial port for our plugged in Arduino board. You’ll need to adjust the path to the TTY device depending on how the Arduino gets enumerated. We are also storing the data for Node-RED (including flows) in the Libreelec storage directory, so that we can re-create the container without losing our flows.

Once this starts up, you should be able to access a nice clean Node-RED instance on port 1880 of your Libreelec Pi. At this point it’s probably a good idea secure your instance and set up projects for backing up your flows via Git.

MySensors Proxy Flow

So here’s what you came here for. Using the MySensors nodes I’ve created a flow which receives MySensors data via the serial port decodes it to a Javascript object and then immediately encodes it and sends it via MQTT. To do this we make use of the serial decode and MQTT encode nodes. In the opposite direction we receive MQTT data on the mysensors/in/# wildcard and decode it to another JS object. This is then immediately encoded for serial and sent out the serial port. The finished flow looks like this:

mysensors node red proxy flow
The MySensors proxy flow

The full JSON for this is below. You’ll need to update the details for the MQTT broker and serial port to match your installation:

[{"id":"750924cf.e87e14","type":"tab","label":"MySensors MQTT Proxy","disabled":false,"info":""},{"id":"e7a50a55.c700d8","type":"mysdecode","z":"750924cf.e87e14","database":"","name":"Decode Serial","mqtt":false,"enrich":false,"x":244,"y":125,"wires":[["1fe04ed1.423d09"]]},{"id":"1fe04ed1.423d09","type":"mysencode","z":"750924cf.e87e14","name":"Encode MQTT","mqtt":true,"mqtttopic":"mysensors/out","x":429,"y":125,"wires":[["73eb8bcf.3be6bc"]]},{"id":"73eb8bcf.3be6bc","type":"mqtt out","z":"750924cf.e87e14","name":"","topic":"","qos":"","retain":"","broker":"61f77eeb.b75e2","x":584,"y":125,"wires":[]},{"id":"1a5ca295.7db125","type":"serial in","z":"750924cf.e87e14","name":"serial in","serial":"31aea0d6.e5a438","x":87,"y":125,"wires":[["e7a50a55.c700d8"]]},{"id":"5a8336a4.b9d398","type":"mqtt in","z":"750924cf.e87e14","name":"","topic":"mysensors/in/#","qos":"2","datatype":"auto","broker":"61f77eeb.b75e2","x":117,"y":179,"wires":[["b8c6fd6f.3ecf08"]]},{"id":"b8c6fd6f.3ecf08","type":"mysdecode","z":"750924cf.e87e14","database":"","name":"Decode MQTT","mqtt":true,"enrich":false,"x":320,"y":180,"wires":[["440bfad4.44eadc"]]},{"id":"440bfad4.44eadc","type":"mysencode","z":"750924cf.e87e14","name":"Encode Serial","mqtt":false,"mqtttopic":"","x":508,"y":179,"wires":[["e8c5f4bd.bc3d3"]]},{"id":"e8c5f4bd.bc3d3","type":"serial out","z":"750924cf.e87e14","name":"serial out","serial":"31aea0d6.e5a438","x":676,"y":179,"wires":[]},{"id":"61f77eeb.b75e2","type":"mqtt-broker","z":"","name":"Home Broker","broker":"mqtt.example.com","port":"1883","tls":"b8f1023d.4df4","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"nodered/status","birthQos":"2","birthRetain":"true","birthPayload":"online","closeTopic":"nodered/status","closeQos":"2","closeRetain":"true","closePayload":"offline","willTopic":"nodered/status","willQos":"2","willRetain":"true","willPayload":"offline"},{"id":"31aea0d6.e5a438","type":"serial-port","z":"","serialport":"/dev/ttyUSB0","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","waitfor":"","dtr":"none","rts":"none","cts":"none","dsr":"none","newline":"\\n","bin":"false","out":"char","addchar":"","responsetimeout":"10000"},{"id":"b8f1023d.4df4","type":"tls-config","z":"","name":"Home Broker","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","verifyservercert":true}]

That’s it for the flow. We can integrate this into HASS by changing our previous gateway configuration to use MQTT:

mysensors:
  gateways:
    - device: mqtt
      persistence_file: '/config/mysensors.json'
      topic_in_prefix: 'mysensors/out'
      topic_out_prefix: 'mysensors/in'

Conclusion

With this in place I’m now able to receive data from my sensor in the greenhouse without the reception issues I was experiencing. I like the use of Node-RED for these kinds of protocol conversion jobs. Its dataflow model is supremely suited to these operations.

Other than the range issue, the MySensors node I built in my previous post has been pretty reliable. I did make a minor hardware change to the setup I described earlier, which was to move positive battery connection from the raw power line to the VCC line of the Arduino. This is because the raw line is regulated to 3.3V. The voltage regulator will consume some power and also won’t work once the battery voltage drops below about 3.3V. Connecting to VCC powers the micro directly from the battery without the extra power loss. Since this modification the sensor has been running flawlessly for several weeks.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

home assistant rubbish collection

Quick Project: Follow up to my Home Assistant Rubbish Collection Panel

This post may contain affiliate links. Please see the disclaimer for more information.

Last month, I wrote a quick post about the Home Assistant Rubbish Collection panel I made for the Lovelace UI. Well, it looks like amaximus was inspired to create his own custom card to do a similar thing. [NOTE: the author of this card hasn’t contacted me directly (I came across it via HACS), I’m only claiming to be the inspiration based on the relative dates].

This card has a couple of cool capabilities that my previous panel didn’t have. Specifically, you can set colour coded icons for your different bins and the icon will change style and colour (to red) when the bin is due to go out in the next day. You can also hide the card completely if a bin is not due to go out within X days.

home assistant rubbish collection
Cards for all four of my sensors, with colour coded icons
home assistant rubbish collection
If the collection is the next day the icons change to red

Installation and Setup

I installed the card by adding a git submodule to my configuration repository in the www/plugins directory, but you can also install directly from HACS. I’m switching over to adding all my custom components and cards as submodules in order to make my config more easily deployable.

After installation, you need to add the path to the garbage-collection-card.js file in the resources section of your Lovelace UI config:

resources:
  - type: js
    url: /local/plugins/garbage-collection-card/garbage-collection-card.js

Once that’s done you can add cards to the UI. I just put mine in a vertical stack to group them together:

type: vertical-stack
cards:
  - type: 'custom:garbage-collection-card'
    entity: sensor.food_scraps
    hide_date: true
    icon_color: green
    icon_size: 48px
  - type: 'custom:garbage-collection-card'
    entity: sensor.general_recycling
    hide_date: true
    icon_color: yellow
    icon_size: 48px
  - type: 'custom:garbage-collection-card'
    entity: sensor.glass_recycling
    hide_date: true
    icon_color: blue
    icon_size: 48px  
  - type: 'custom:garbage-collection-card'
    entity: sensor.landfill
    hide_date: true
    icon_color: red
    icon_size: 48px

…and that’s it! If you want to hide a card for bins that don’t need to go out soon use hide_before: x (where x is the number of days). I’ll probably use this to hide bins that don’t need to go out in the current week, but I wanted to show all the cards in the screenshots 😉

Conclusion

I think this is a great improvement on my previous panel, so I’m going to stick with it. Thanks to the author another contributors for taking the time to make it!

It’s kinda cool to think that this blog may have inspired someone else to go out and write some code! If you are inspired to make and share something as a result of one of my posts, please get in contact! Your work will most likely get featured in a future blog post.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

home assistant rubbish collection

Quick Project: Rubbish Collection Panel for Home Assistant

This post may contain affiliate links. Please see the disclaimer for more information.

With my local council rolling out an increasingly complex system of rubbish bins and collection, I’ve been thinking about getting this integrated with Home Assistant so that we don’t have to remember which bins go out when. I was presently surprised to find the garbage_collection custom component whilst browsing HACS the other day. This component does exactly what I need, so I decided to give it a go.

Rubbish Collection Configuration

After installing the component via HACS, I set about configuring it. Here is the configuration I came up with:

sensor:
  - platform: garbage_collection
    name: Food Scraps
    frequency: "weekly"
    collection_days: !secret rubbish_collection_day
  - platform: garbage_collection
    name: General Recycling
    frequency: "odd-weeks"
    collection_days: !secret rubbish_collection_day
  - platform: garbage_collection
    name: Glass Recycling
    frequency: "even-weeks"
    collection_days: !secret rubbish_collection_day
  - platform: garbage_collection
    name: Landfill
    frequency: "even-weeks"
    collection_days: !secret rubbish_collection_day

That was easy! The only non-obvious thing is working out which bins are collected on even and odd weeks, which is easy to look up online.

Getting it into Lovelace

The garbage_collection component page in HACS has a nice screenshot of the sensors in Lovelace (which doesn’t seem to be in the repository readme). However, the sensors themselves have a state based on whether the bin is due to be put out. The state is nice and machine readable, but I wanted to recreate the panel from the screenshot for the humans that have to look at it. In the end I actually decided to simplify it down to just show “today”, “this week” or “next week” plus the number of days, since the actual date is pretty irrelevant.

home assistant rubbish collection
The finished rubbish collection panel in my Lovelace UI

This proved to be more difficult than I’d expected, since Lovelace doesn’t support templates in cards natively. I had to install the Lovelace Card Templater plugin via HACS. This plugin in turn requires the card-tools plugin, which I couldn’t find in HACS. I ended up installing it by adding it as a git submodule to my configuration repository. I then added the following to my Lovelace config to load the plugins:

resources:
  - type: js
    url: /local/plugins/lovelace-card-tools/card-tools.js?v=1
  - type: js
    url: /community_plugin/lovelace-card-templater/lovelace-card-templater.js

Full Panel YAML

The panel itself is made up of a vertical stack card in which I put two horizontal stack cards. These in turn contain two of the templater cards. The configuration of the templater cards is a little involved since you need to specify the entity twice (which seems to be due to some internal limitation of Lovelace). My template cards are based on the sensor card to show just the data from the template. I use a state_template to do this.

Anyway, here’s the full YAML:

cards:
  - cards:
      - card:
          entity: sensor.food_scraps
          type: sensor
        entities:
          - entity: sensor.food_scraps
            state_template: >-
              {% if state_attr("sensor.food_scraps", "days") == 0 %} Today {%
              elif state_attr("sensor.food_scraps", "days") < 7 %} This Week {%
              else %} Next Week {% endif %} ({{ state_attr("sensor.food_scraps",
              "days") }} days)
        type: 'custom:card-templater'
      - card:
          entity: sensor.general_recycling
          type: sensor
        entities:
          - entity: sensor.general_recycling
            state_template: >-
              {% if state_attr("sensor.general_recycling", "days") == 0 %} Today
              {% elif state_attr("sensor.general_recycling", "days") < 7 %} This
              Week {% else %} Next Week {% endif %} ({{
              state_attr("sensor.general_recycling", "days") }} days)
        type: 'custom:card-templater'
    type: horizontal-stack
  - cards:
      - card:
          entity: sensor.glass_recycling
          type: sensor
        entities:
          - entity: sensor.glass_recycling
            state_template: >-
              {% if state_attr("sensor.glass_recycling", "days") == 0 %} Today
              {% elif state_attr("sensor.glass_recycling", "days") < 7 %} This
              Week {% else %} Next Week {% endif %} ({{
              state_attr("sensor.glass_recycling", "days") }} days)
        type: 'custom:card-templater'
      - card:
          entity: sensor.landfill
          type: sensor
        entities:
          - entity: sensor.landfill
            state_template: >-
              {% if state_attr("sensor.landfill", "days") == 0 %} Today {% elif
              state_attr("sensor.landfill", "days") < 7 %} This Week {% else %}
              Next Week {% endif %} ({{ state_attr("sensor.landfill", "days") }}
              days)
        type: 'custom:card-templater'
    type: horizontal-stack
type: vertical-stack

Each of the templater cards is pretty much the same. I just change the entity ID for each. The only improvement I would make would be to add a line break between the “Today”/”This week”/Next Week” text and the day count, since this would look slightly better. I couldn’t work out how to do that however.

Conclusion

I think I’ve achieved my goal of simplifying the task of remembering which bins go out when. Now I can quickly check that info with a glance at my HASS UI. Of course, now that I have the rubbish collection data in Home Assistant I can use it for other things such as notifications or reminder lights. I already have some ideas for status lighting, so that might become part of a larger project.

I’d like to say thanks to the authors of the custom components and plugins that I’ve used to achieve this. The HASS community really is thriving will all these third party addons at the moment!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.