Remote Workstation with x2go and Manjaro Linux

February 12, 2020Rob Connolly Leave a comment

This post may contain affiliate links. Please see the disclaimer for more information.

I’ve been running a very under-powered and increasingly ancient laptop, a ThinkPad X131e, for several years. It’s been upgraded over time with an SSD and a replacement battery. For my requirements this is mostly OK, since my workload mostly consists of a web browser and terminal windows and it runs my preferred desktop (KDE Plasma Desktop) just fine. Since my server performs the majority of my computing I haven’t really been limited by this. However, every now and then I need to run a couple of apps which just bring it to a halt as it thrashes around trying to swap. Clearly 4GB of RAM is not enough to run modern applications.

I actually do have another desktop computer, but my use of it recently has been limited to the odd DVD rip. This was mainly due to having to switch the mouse, keyboard and monitor over from my work machine in order to use it in the limited space which is my home office. I could have probably remedied this with a KVM switch, but somehow I never got around to it. I was also not enthused about spending my evenings sitting in the same office I’ve just been working in all day. So that computer sat gathering dust for the most part.

That was until I came across a post about using x2go to create a remote workstation after musing whether this would be possible. I decided to do the same, since I had the hardware already available to make it work.

A (Semi) Failed Experiment

Initially I did a few tests with local VMs to see if x2go was going to work with KDE, since I’d seen mixed reports about this. The good news is that it works pretty well (at least for basic remote desktop, I’ll come to some of the problems below). The bad news is that my preferred desktop distro – KDE Neon – didn’t work well. First of all I couldn’t install the client on my laptop due to a dependency issue in APT. Secondly, although the desktop worked fine I was unable to suspend the session due to some Systemd/D-BUS issue. So I tried another distro. I’d heard good things about Manjaro and the KDE edition works great with x2go (both client and server). It also has a really nice default theme!

I also wanted the remote desktop to run as a VM on the machine, under Proxmox, so that I could potentially switch distros easily or create extra VMs for other purposes. I spent quite a bit of time configuring this only to find a few issues. The first was that I couldn’t get the host machine to pass through the internal DVD drive to the VM which was a deal breaker. The second was that suspending the VM and shutting down the host was pretty clunky and prone to just hang for no reason. As I added this machine in a cluster with my other host, the cluster would also lose quorum when the host went down. This causes lots of things to fail, including VM backups on the remaining host.

Back to Bare Metal

I decided to abandon the VM approach for now and go with a bare metal install to see if I can work with the remote desktop system. It wasn’t a complete loss, since I got a chance to try out the newly updated clustering in Proxmox, which will be relevant when I convert my existing Ubuntu server over.

The bare metal install of Manjaro was pretty boring (which is a good thing, installing a Linux distro should be boring and stable!). One thing I noticed was that I wasn’t able to manually set up LVM from the GUI installer. I could create volume groups, but it wouldn’t let me add partitions to them! As far as I understand Manjaro Architect lets you do this. I played with this a bit later when installing Manjaro on my Laptop, but opted to go with the default encrypted system option from the GUI installer. If I ever come to reinstall on the desktop machine, I’ll look at Manjaro Architect further.

Saving Power

Since this machine won’t be in use all the time, I wanted to shut it down and power it up remotely. I also wanted my x2go session to be persistent so I could pick up where I left off. For this reason I opted to use hibernate coupled with wake-on-LAN.

Configuring the wake-on-LAN took a little while. Even after enabling it in the BIOS and configuring network manager, it still didn’t work. It turned out that it was being disabled by TLP. After fixing that it worked fine.

The next problem was that I wanted to power the system up and down via a switch in Home Assistant. This was difficult as the Home Automation system is on a different subnet to the machine in question. I opted in the end to run the WOL command from my pfSense firewall over SSH. The following HASS configuration gave me the switch I was looking for:

shell_command:
  desktop_power_on: "ssh -i /config/id_rsa -o 'StrictHostKeyChecking=no' home-assistant@<my_firewall> -- '-i <subnet broadcast address> <mac address>'"
  desktop_power_off: "ssh -i /config/id_rsa -o 'StrictHostKeyChecking=no' <user>@<my_desktop>"

binary_sensor:
  - platform: ping
    name: "Desktop Computer State"
    host: <desktop ip>
    count: 1
    scan_interval: 5

switch:
  - platform: template
    switches:
      desktop_computer_power:
        value_template: "{{ is_state('binary_sensor.desktop_computer_state', 'on') }}"
        turn_on:
          service: shell_command.desktop_power_on
        turn_off:
          service: shell_command.desktop_power_off

This requires a bit of setup on both the firewall and the desktop machines. First I created a user in pfSense with the relevant permissions to log in via SSH. I then added the following to the authorised SSH keys field:

command="wol $SSH_ORIGINAL_COMMAND",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-rsa .....

This basically allows the SSH key to only run the wol command and to pass through the original command as arguments to it. You’ll note that in the power on command above, only the arguments are specified. This means that I can send WOL packets to any machine on the network, but the key can’t do anything else.

There is a similar bit of configuration on the desktop machine:

command="systemctl hibernate",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-rsa ....

This restricts the key to only running the systemctl hibernate command.

With that in place I have a nice switch in my HASS GUI to power up and down the machine. I can also automate it to power up and down under certain conditions if I wish.

Setting Up x2go

Since Manjaro is based on Arch Linux, I just installed the x2goserver package on my desktop and the x2goclient package on my laptop.

x2go is fairly trivial to set up, requiring only the enabling of X11 forwarding in the SSH daemon on the server side, just follow the instructions to do this.

In order to connect to a KDE desktop running on the server, we need to set up a profile in the x2go client. The main thing here is to set the session type to “Custom desktop” with the command startplasma-x11 (the KDE session type doesn’t work for some reason). Obviously you also need to set the address of the machine to log in to. I also found that I needed to set the path to my SSH key in order to have it be used by the client.

The Finished Product

The final product is pretty nice. I can remotely boot my desktop machine from my laptop or phone. I then connect with x2go from my laptop and get to work! My previous session will be restored if there is one, meaning I can just pick up where I left off. The connection to the remote system is excellent, with no noticeable delay. So far I’ve only tested this over my local wifi, but that will be 97% of it’s usage anyway. There is some tearing when moving windows and some graphical issues when the x2go session window gets resized, but these resolve themselves after a few seconds.

One wrinkle is shared folders, which are supposed to be supported by x2go, but currently seem to be broken. I get an error message, similar to that described here when I try to mount one. Apparently that bug is fixed, but I guess the version containing the fix hasn’t been released yet. If the new version doesn’t land soon I’ll probably try and work around it with SSHFS+some scripting. For now it’s not too much of an issue, all my files are now on the desktop anyway!

I also don’t see any sound devices on the remote system, but I haven’t tried playing sound to see if it’s working. I tend not to listen to music or watch movies on my laptop anyway and I can still do this locally if need be.

Conclusion

Overall, I’m pretty happy with this setup. It’s not perfect, but it is nice to have easier access to my other machine. I still have a lot of setup to do at both ends to make this work better and to make the remote machine feel like home, but it’s getting there.

I’m not sure if I’ll run this setup full time yet or just for certain applications. I’ve decided to upgrade the RAM in both systems as well as my server, so we’ll see how the balance of local vs remote falls out after doing that! I’m also eyeing a Pinebook Pro as a potential replacement laptop.

I really like Manjaro as a distro. It’s not quite as polished a KDE experience as KDE Neon is, but it’s getting there. I like the Arch base – I’ve been a fan of Arch since my very first experience with it (CAUTION: very old post!), but I just couldn’t get on with it as a main driver. Manjaro gives you a nice experience out of the box with the power to tweak as much as you like.

I’ll be sure to update on any further progress with fixing some of the issues I’ve encountered with this project. In the meantime, if anyone has solutions/workarounds to the shared folder issue, please leave a comment. Bye for now!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

Keeping Baddies Away with Occupancy Simulation and Home Assistant

January 29, 2020Rob Connolly 1 Comment

This post may contain affiliate links. Please see the disclaimer for more information.

Home security is one of the more interesting and useful applications of home automation technologies. Of-course it’s a whole field and industry in itself with all manner of products and services available. Of course the first part of any security system should be good physical security (good doors, windows and locks). Once you get past this it seems like you can go two ways: prevention or detection.

Most traditional systems are in the detection camp. They aim to detect an intruder and alert someone, either via the traditional loud noise or other means. The presence of an alarm box or sign my provide a minimal amount of prevention capability, but not much. Camera systems on the other hand provide both pretty robust prevention as well as (increasingly) advanced detection capabilities.

The third option is to ramp up the prevention angle by making it look like someone is home. It’s this that I’m going to tackle here, with an occupancy simulation application for Home Assistant called Occusim.

Note: I’m not a home security specialist and you should definitely consult a professional. This post is provided for informational purposes and should not be relied upon to provide a full security system. The author excepts no responsibility for any consequence of you relying on this information and/or not consulting someone who actually does this stuff for a living. YOU HAVE BEEN WARNED!

What is Occupancy Simulation?

Basically it’s pretending you are home, by switching on and off things as you would when you’re there. The reasoning behind it is that potential intruders are less likely to invade a house with people inside. I’m sure there are parts of the world where that doesn’t hold. However, I’d guess it applies in enough places to be useful.

Methods of occupancy simulation range from the very low tech “light on a timer” to the high tech approach we’re going to take here, using Home Assistant. The key is to make the sequence as close to the normal usage pattern as possible. So that anyone potentially watching your house can’t tell the difference.

Introducing OccuSim

OccuSim is a tool for defining complex occupancy simulation rules, for use with Home Assistant. These rules are defined as sequences which proceed from beginning to end (with potentially some randomness applied). In this way you can program in a full day’s operation. There is also a fully random mode, which works better for non-scheduled events. Both modes can be combined to give a good representation of your normal home operation.

OccuSim is actually an AppDaemon app, so you will need AppDaemon installed as a dependency. I’m not going to cover that now. You can either see the link above or read my previous post on the subject.

Once you have a functional AppDaemon install you can install OccuSim by cloning the repository into the apps subdirectory of you AppDaemon configuration. Rather than just a pure clone, I prefer to add it as a submodule in my AppDaemon configuration repository. This makes it easy to reinstall if you move your configuration around:

cd apps
git submodule add https://github.com/acockburn/occusim.git occusim
git commit -m "Add OccuSim"

With this setup, updating OccuSim is a little more involved. This is because you also need to commit the submodule change:

cd apps/occusim
git pull origin
cd ../..
git add apps/occusim
git commit -m "Update OccuSim"

Initial Configuration

OccuSim is configured as an app in your apps.yaml file. There is some initial config before you get down to creating your simulation sequences:

occupancy_simulator:
  class: OccuSim
  module: occusim
  log: '1'
  notify: '1'
  enable: input_boolean.vacation_mode,on
  test: '0'
  dump_times: '1'
  reset_time: '02:00:00'

Here I add a new app to my setup, called occupancy_simulator. We set the Python class to load as OccuSim from the module occusim, since we installed it in it’s own subdirectory. I set OccuSim to log it’s activity and to notify when a sequence step is activated. This will use the default notify.notify service in HASS, so you better have that set up to go to the right place. I haven’t found a way to change the notifier that it uses.

The enable setting takes an input_boolean entity and a state in which OccuSim should be active. Here I use my vacation mode toggle. I actuate mine manually, but it’s perfectly possible to set this from a HASS automation if you so desire.

The last few settings are some basic housekeeping. Test mode is disabled to ensure that OccuSim does it’s thing (though this can be useful when debugging your sequences). I also set the dump_times option to true so that I can see the times of the steps in the logs. I then set the time when I want to re-calculate the steps for the upcoming day. In my case this is set to 2am.

Setting up Sequences

A simple sequence might be the following:

step_evening_name: Evening
step_evening_start: 'sunset'
step_evening_on_1: scene.main_lights

This sequence simply turns on my main lights scene at sunset. The configuration variables take the form step_<id>_<variable> where <id> is a custom identifier that needs to be common across all variables for the step. Multiple entities can be turned on or off by appending numbers to the variables in question, e.g. step_evening_on_1, step_evening_on_2, etc.

You can add an offset to the times for sunset or sunrise, such as sunset - 00:20:00 to trigger 20 minutes before sunset. You can of course also specify an absolute time in the form HH:MM:SS.

More Advanced Sequences

So far that’s great, but we could have just turned on a light at sunset with a basic HASS automation rule. The real power of OccuSim comes in randomising the sequence times (within bounds) and creating multi-step sequences.

Expanding on our previous example:

step_evening_name: Evening
step_evening_start: 'sunset - 00:40:00'
step_evening_end: 'sunset - 00:10:00'
step_evening_on_1: scene.main_lights

This will turn on the lights at a random time between 40 minutes before sunset and ten minutes before sunset.

Let’s now create a multi-step sequence:

step_movie1_name: Movie Scene
step_movie1_start: '20:00:00'
step_movie1_end: '20:30:00'
step_movie1_on_1: scene.movie

step_movie2_name: Movie Scene Pause
step_movie2_relative: Movie Scene
step_movie2_start_offset: '00:35:00'
step_movie2_end_offset: '00:45:00'
step_movie2_on_1: script.downlights_bright

step_movie3_name: Movie Scene Play
step_movie3_relative: Movie Scene Pause
step_movie3_start_offset: '00:03:00'
step_movie3_end_offset: '00:06:00'
step_movie3_on_1: scene.movie

Here I start a sequence that will execute my movie scene sometime between 8pm and 8.30pm. I then specify a second step which will emulate us pausing the movie and putting the kitchen lights on. This happens sometime between 35 and 45 minutes later and is relative to the previous step. This means that whatever time the previous step is executed the second step will always come 35-45 minutes after that. The third step in the sequence is another relative one. This will execute 3-6 minutes after the previous one and emulates us starting up the movie again.

Random Mode

As mentioned earlier, you can also create totally random events. For example you might use the following to simulate overnight bathroom trips:

random_bathroom_name: Overnight Bathroom
random_bathroom_start: Bedtime
random_bathroom_end: Morning
random_bathroom_minduration: 00:02:00
random_bathroom_maxduration: 00:05:00
random_bathroom_number: 2
random_bathroom_on_1: light.bathroom
random_bathroom_off_1: light.bathroom

step_bedtime_name: Bedtime
step_bedtime_start: '21:30:00'
step_bedtime_end: '22:30:00'

step_morning_name: Morning
step_morning_start: 'sunrise + 00:20:00'

This configuration creates a 2 randomised events lasting 2-5 minutes, which turn on and off the bathroom light. These are defined as starting and ending relative to other steps in the sequence. I’ve defined the Bedtime and Morning steps in my example to illustrate this. These steps can of course contain actions of their own just like the steps above. The full configuration basically says “sometime starting between 9.30pm and 10.30pm and lasting until 20 minutes after sunrise there will be two instances of the bathroom light coming on for 2-5 minutes”. Pretty cool!

Conclusion

OccuSim is a pretty powerful tool, which allows you to create complex occupancy simulation rules with Home Assistant. I’ve only really just begun to map out my own simulations. You can find these on GitLab. Some of the examples here are based upon my rules, but the real times have been changed. I’m going to continue expanding upon this in the coming weeks. Hopefully I’ll end up with a pretty accurate simulation.

I’m also exploring further home security options and will hopefully be expanding my existing ZoneMinder setup this year. Stay tuned to the blog for more info when that happens.

Smart Outdoor Solar Powered Lighting with the ESP32

January 14, 2020Rob Connolly 7 Comments

This post may contain affiliate links. Please see the disclaimer for more information.

The primary form of heating in our home is a wood burning stove. You’re probably thinking that this isn’t very ‘smart’ and you’d be right. It’s the least smart part of my smarthome, requiring daily lighting and monitoring during the winter months. What it is though is eco-friendy and cheap to run. Also, nothing quite beats a wood fire for warmth and hygge. The point of all this is that we have a wood shed (two actually) at the back of our house. During dark winter nights, trips to these have been fraught with danger and mystery due to a lack of lighting. That is until now: this winter my completely over-engineered, ESP32 driven, solar powered, smart LED lights will light the way!

Of course, I could just use a head torch for my after dark trips to the wood shed. Or buy any number of (doubtless cheap and nasty) solar powered lights from the local DIY store. However, that didn’t seem very fun and wouldn’t integrate with Home Assistant and the rest of my smarthome.

Components

For these lights I decided to use WS2812B light strip driven by an ESP32. The solar power system would utilise an off the shelf solar controller and lead acid battery along with a 5W solar panel which I acquired cheap from work.

Full Parts List

An ESP32 development board (I used the DOIT board, but pretty much any ESP32 board will do);
1x 5A adjustable power supply, which will be set to 5V for the LED strip;
2m of IP67 rated WS2812B light strip;
1x single channel relay board;
1x 20A solar controller;
5W solar panel (since I acquired this from work I don’t have a specific panel to recommend);
Various resistors for the power monitoring circuits;
A DHT22/AM2302 temperature/humidity sensor;
IP65 rated electronics case (this was mine, but you should search around to find one that will fit your electronics);
1x IP68 PG9 cable gland (comes as a pack of 10, but very useful to have around)
1x 7Ah lead acid battery (I acquired mine from a local supplier, so I don’t have a link to share);
10-15 of these 3D printed U-brackets (based on this design);
A larger U-bracket for holding the battery (I actually printed this slightly too large and had to trim down the sides);
Various screws, cable ties, double sided tape and heatshrink tubing;
Clear roofing sealant for sealing cable joins;
Several meters of speaker wire.

solar powered esp32 panel specs — The specs of the solar panel

GCode for the Ender 3 is also available for the 3D printed parts. If you have another printer you’ll need to slice the STL files yourself. I had a few issues with some of the small brackets adhering to the bed, but I was able to get a high enough yield thanks to the OctoPrint-Cancelobject plugin.

The charge controller is rather over-spec’d for the LEDs I’m actually running (which pull less than 2A at the 12V battery voltage, on full brightness). This is because I originally thought I would use the full 5m of LED strip that I ordered. However when it arrived and I saw how bright they are I downgraded to just 2m. I had also ordered a beefier power supply for this purpose. The one I ended up using was left over from my failed tablet salvaging efforts. This means I have 3m of LED strip and a decent power supply left over for a future project.

Putting it Together

I spent the better part of a day putting together the components and soldering all the cabling. I then spent at least another day mounting things on the shed and fixing issues. All the joins between cables are double covered with heatshrink tubing. The first layer insulates the cable from any neighbouring cables. The second layer is for waterproofing and has roof sealant injected into it. The same is done on the joins between the cable and the LED strip. Hopefully this will stand up to the intense New Zealand rain. Seriously, you have not seen rain until you’ve seen NZ rain! Luckily most of the components are mounted out of the worst of the weather. The cables enter the box through the IP68 cable gland which should be weather tight.

solar powered esp32 mounting — Mounting the electronics in the case

mounting battery — The battery is held securely with that bracket

led mounting bracket — Mounting the first LED bracket

led strip mounted — The LED strip mounted to the shed

When I tested the LEDs after soldering the lead out wires to them I found that the colours were off and transitions and effects were flickery. This could have been due to either a power issue or an issue on the data line. Measuring the 5V line showed minimal voltage drop there. Since the LEDs were OK before adding the lead out I went with the data issue.

solar powered esp32 sacrificial pixel — The sacrificial pixel assembly before mounting in the case

The issue turned out to be due to voltage drop on the 3.3V signal from the ESP32 along the lead out cable. The data line is quite sensitive to voltage drop here because the LEDs are supposed to receive a 5V data signal. They work with 3.3V, but not much lower. In order to solve this I added an single extra pixel at the microcontroller end to boost the signal voltage from 3.3V to 5V. This solves the issue since the data signal is being amplified by each pixel in the chain.

ESPHome Code

The ESPHome code to drive the lights is fairly simple. First we start with the standard setup:

---
esphome:
  name: shed_lights
  platform: ESP32
  board: esp32doit-devkit-v1

wifi:
  ssid: !secret wifi_ssid2
  password: !secret wifi_passwd2
  use_address: !secret shed_lights_ip

mqtt:
  broker: !secret mqtt_broker
  username: !secret mqtt_user
  password: !secret mqtt_passwd

# Enable logging
logger:

ota:
  password: !secret ota_passwd

Here we define the board type, set up the wifi and MQTT connections, enable logging and set up OTA updates. If you’re wondering why I’m using MQTT rather than the ESPHome API, it’s for no other reason than I like MQTT!

power_supply:
  - id: 'led_power'
    pin:
      number: GPIO25
      inverted: true

Next I set up a power supply component. This is used along with the single channel relay to automatically power up and down the power supply to the LEDs. This will save a bit of power and also makes sure that there is no power flowing in the cables outside the box for most of the time, which may help in the event of a leaky connection. In order to do this I’m not running the ESP32 from the same 5V supply, instead using one of the USB ports from the solar controller.

The LED strip configuration is then pretty standard:

light:
  - platform: fastled_clockless
    chipset: WS2812B
    pin: GPIO23
    num_leds: 61
    rgb_order: GRB
    name: "Shed Lights"
    effects:
      - addressable_rainbow:
      - addressable_color_wipe:
      - addressable_scan:
      - addressable_twinkle:
      - addressable_random_twinkle:
      - addressable_fireworks:
      - addressable_flicker:
    power_supply: 'led_power'

Note that there are 61 pixels here, that 2m at 30 pixels per meter plus one sacrificial voltage boosting pixel. The addition of the effects is a bit of a gimmick since I’m mostly interested in white light for the application. I only bought the RGB LEDs because the price difference wasn’t enough to justify only buying white.

solar powered esp32 finished — The finished electronics mounted to the shed

solar powered esp32 panel — The solar panel is mounted to the fence behind the shed

solar powered esp32 relative positions — The electronics box and panel on the end of the shed

Voltage Sensing and Health Monitoring

After putting all this together and mounting it on the shed I decided that I’d like to have some form of monitoring for the voltages from the battery and solar panel. The solar controller obviously monitors these but there is no way to get this data out.

In the end I soldered up a couple of voltage divider circuits and added these to the setup in the box. I also added a DHT22 sensor for temperature and humidity sensing inside the box.

voltage monitoring schematic — Schematic of the voltage dividers used for the power monitoring circuits

The ESPHome configuration for these follows. I also added a binary status sensor and a WiFi signal sensor to allow me to monitor the system remotely.

binary_sensor:
  - platform: status
    name: "Shed Lights Status"

sensor:
  - platform: wifi_signal
    name: "Shed WiFi Signal"
    update_interval: 180s
  - platform: dht
    pin: GPIO15
    model: AM2302
    temperature:
      name: "Shed Battery Box Temperature"
    humidity:
      name: "Shed Battery Box Humidity"
    update_interval: 180s
  - platform: adc
    pin: GPIO39
    name: "Shed Battery Voltage"
    icon: "mdi:car-battery"
    attenuation: "11db"
    filters:
      - multiply: 4.24
      - sliding_window_moving_average:
          window_size: 12
          send_every: 12
    update_interval: 15s
  - platform: adc
    pin: GPIO36
    name: "Shed Solar Panel Voltage"
    icon: "mdi:solar-panel"
    attenuation: "11db"
    filters:
      - multiply: 5.73
      - filter_out: 0.00
      - sliding_window_moving_average:
          window_size: 12
          send_every: 12
    update_interval: 15s

The voltage readings I am getting from the two voltage sensors are a little weird. The battery voltage is higher than I would expect and the solar panel voltage is lower. I double checked the multiplication factors against the raw ADC readings and the resistors used and the readings make sense. Initially I thought this could be due to the temperature inside the box (~50Â°C when it’s in full sun!), but now I’m not so sure since this is also the case at lower temperatures. It could be the behaviour of the charge controller. I’ll continue to monitor it over different charge states.

shed lights hass card — The card in Home Assistant showing the system status and lighting controls

The Finished Product

Now the moment you’ve all been waiting for – gratuitous photos of the LEDs in fancy colours!

shed lights white — In white, the intended operation mode

shed lights red — Red is insanely bright!

shed lights purple — Purple/pink, the favourite of some people in the household

Conclusion

This is actually my first time using the WS2812B LED strip and I have so say I’m really impressed. You can be sure there will be other LED lighting projects coming in future now that I’ve dipped my toes in!

I’m really pleased with the final product. The LEDs look awesome and provide ample light for their task. The ESPHome base has so far been rock solid in terms of stability, which is what I’ve come to expect from using it in other projects.

The initial intention was to make these lights motion activated. However, I couldn’t find a motion sensor which was suitable for outdoor use. I’d also have to locate the sensor at the other end of the sheds from the battery box which would mean a whole load of wiring. As such I’ve decided to build by own wireless outdoor motion sensor and connect it to my MySensors network. I’ll then trigger the LEDs via an automation in Home Assistant. I’ll post an update on this when I have it running.

This has been a really fun and interesting project. As always, please let me know what you think in the feedback channels and feel free to share your own LED lighting projects.

Quick Project: MySensors MQTT Proxy with Node-RED

January 10, 2020Rob Connolly 3 Comments

This post may contain affiliate links. Please see the disclaimer for more information.

In my previous exploration of MySensors, I encountered issues with the range of the nrf24l01+ radio. One of the simplest solutions to this problem is to move the gateway closer to the intended deployment of the sensors. In my case the sensor is positioned on the back wall of our house, just outside the master bedroom. Connected to the TV inside the master bedroom is a Raspberry Pi running Libreelec. I reasoned that I should be able to use this Pi to proxy the MySensors serial data to Home Assistant via MQTT, using the MySensors Node-RED nodes. With this approach I can make a new MQTT gateway using hardware I already have deployed.

It was also suggested that adding a 10-100uF 6.3/10V capacitor across the power line to the radio would also help with the range issues. I’m definitely going to do this, but the components haven’t arrived yet!

Hardware Setup

If you followed my previous post all you need to do here is plug in the serial gateway to the Pi! Otherwise you’re going to need to build a gateway (and maybe some sensors). Go check out my original post for more details.

It’s probably worth going through how this is going to work. Basically, the idea is that the RF data will be received by our existing MySensors gateway and sent via serial to the RPi. This will be read by Node-RED, re-encoded and sent via MQTT to Home Assistant which will still act as the MySensors controller:

Serial GW â‡’ Serial Connection â‡’ Node-RED (on RPi) â‡’ MQTT â‡’ HASS (Controller)

In the return direction this dataflow operates in reverse:

HASS â‡’ MQTT â‡’ Node-RED â‡’ Serial Connection â‡’ Serial GW

Deploying Node-RED on Libreelec

Libreelec is a cut down appliance-like Linux distribution, specifically for running Kodi. At first glance it seems like installing a third party program such as Node-RED would be difficult. However, Libreelec has Docker support via an add-on, which makes things as easy as running the command:

$ docker run -d --restart=always --device=/dev/ttyUSB0:/dev/ttyUSB0 --group-add dialout -v /storage/node-red-data:/data -p 1880:1880 --name node-red nodered/node-red:latest

This starts the Node-RED Docker container with access to the serial port for our plugged in Arduino board. You’ll need to adjust the path to the TTY device depending on how the Arduino gets enumerated. We are also storing the data for Node-RED (including flows) in the Libreelec storage directory, so that we can re-create the container without losing our flows.

Once this starts up, you should be able to access a nice clean Node-RED instance on port 1880 of your Libreelec Pi. At this point it’s probably a good idea secure your instance and set up projects for backing up your flows via Git.

MySensors Proxy Flow

So here’s what you came here for. Using the MySensors nodes I’ve created a flow which receives MySensors data via the serial port decodes it to a Javascript object and then immediately encodes it and sends it via MQTT. To do this we make use of the serial decode and MQTT encode nodes. In the opposite direction we receive MQTT data on the mysensors/in/# wildcard and decode it to another JS object. This is then immediately encoded for serial and sent out the serial port. The finished flow looks like this:

The full JSON for this is below. You’ll need to update the details for the MQTT broker and serial port to match your installation:

[{"id":"750924cf.e87e14","type":"tab","label":"MySensors MQTT Proxy","disabled":false,"info":""},{"id":"e7a50a55.c700d8","type":"mysdecode","z":"750924cf.e87e14","database":"","name":"Decode Serial","mqtt":false,"enrich":false,"x":244,"y":125,"wires":[["1fe04ed1.423d09"]]},{"id":"1fe04ed1.423d09","type":"mysencode","z":"750924cf.e87e14","name":"Encode MQTT","mqtt":true,"mqtttopic":"mysensors/out","x":429,"y":125,"wires":[["73eb8bcf.3be6bc"]]},{"id":"73eb8bcf.3be6bc","type":"mqtt out","z":"750924cf.e87e14","name":"","topic":"","qos":"","retain":"","broker":"61f77eeb.b75e2","x":584,"y":125,"wires":[]},{"id":"1a5ca295.7db125","type":"serial in","z":"750924cf.e87e14","name":"serial in","serial":"31aea0d6.e5a438","x":87,"y":125,"wires":[["e7a50a55.c700d8"]]},{"id":"5a8336a4.b9d398","type":"mqtt in","z":"750924cf.e87e14","name":"","topic":"mysensors/in/#","qos":"2","datatype":"auto","broker":"61f77eeb.b75e2","x":117,"y":179,"wires":[["b8c6fd6f.3ecf08"]]},{"id":"b8c6fd6f.3ecf08","type":"mysdecode","z":"750924cf.e87e14","database":"","name":"Decode MQTT","mqtt":true,"enrich":false,"x":320,"y":180,"wires":[["440bfad4.44eadc"]]},{"id":"440bfad4.44eadc","type":"mysencode","z":"750924cf.e87e14","name":"Encode Serial","mqtt":false,"mqtttopic":"","x":508,"y":179,"wires":[["e8c5f4bd.bc3d3"]]},{"id":"e8c5f4bd.bc3d3","type":"serial out","z":"750924cf.e87e14","name":"serial out","serial":"31aea0d6.e5a438","x":676,"y":179,"wires":[]},{"id":"61f77eeb.b75e2","type":"mqtt-broker","z":"","name":"Home Broker","broker":"mqtt.example.com","port":"1883","tls":"b8f1023d.4df4","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"nodered/status","birthQos":"2","birthRetain":"true","birthPayload":"online","closeTopic":"nodered/status","closeQos":"2","closeRetain":"true","closePayload":"offline","willTopic":"nodered/status","willQos":"2","willRetain":"true","willPayload":"offline"},{"id":"31aea0d6.e5a438","type":"serial-port","z":"","serialport":"/dev/ttyUSB0","serialbaud":"115200","databits":"8","parity":"none","stopbits":"1","waitfor":"","dtr":"none","rts":"none","cts":"none","dsr":"none","newline":"\\n","bin":"false","out":"char","addchar":"","responsetimeout":"10000"},{"id":"b8f1023d.4df4","type":"tls-config","z":"","name":"Home Broker","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"","verifyservercert":true}]

That’s it for the flow. We can integrate this into HASS by changing our previous gateway configuration to use MQTT:

mysensors:
  gateways:
    - device: mqtt
      persistence_file: '/config/mysensors.json'
      topic_in_prefix: 'mysensors/out'
      topic_out_prefix: 'mysensors/in'

Conclusion

With this in place I’m now able to receive data from my sensor in the greenhouse without the reception issues I was experiencing. I like the use of Node-RED for these kinds of protocol conversion jobs. Its dataflow model is supremely suited to these operations.

Other than the range issue, the MySensors node I built in my previous post has been pretty reliable. I did make a minor hardware change to the setup I described earlier, which was to move positive battery connection from the raw power line to the VCC line of the Arduino. This is because the raw line is regulated to 3.3V. The voltage regulator will consume some power and also won’t work once the battery voltage drops below about 3.3V. Connecting to VCC powers the micro directly from the battery without the extra power loss. Since this modification the sensor has been running flawlessly for several weeks.

Home Assistant Wall Mounted Tablet Update

January 6, 2020Rob Connolly 5 Comments

This post may contain affiliate links. Please see the disclaimer for more information.

This is an update to my previous post on my wall mounted Home Assistant tablet. There have been quite a few changes, so I thought it was worth another post on the subject. Let’s get into it…

Power Failure

I’ll start with the big issue which has plagued this project – power. Since the battery in the original tablet was completely exhausted I had built a wired power supply to replace it. Unfortunately, this wasn’t powerful enough and the tablet would power off sometimes under high load. This problem seemed to get worse until the tablet would power off after just a few minutes, thus becoming pretty much useless.

I mentioned previously that I had ordered a 5A power supply to fix this problem, but even that didn’t work. The tablet must pull transient currents of more than 5A, which a battery can supply quite easily. Unfortunately the fixed power supply cannot. This is a real shame because I put a lot of effort into the new power supply – I even printed a case for it based on a modified version of the parametric PCB case I used in my recent MySensors build.

Even with the failure of the new power supply, the exercise in building this has demonstrated the utility of a wall mounted tablet in our household. The decision was therefore taken to replace the old broken tablet with a new one.

The New Tablet

The new tablet is the Lenovo Tab E8 (link is to Amazon, but ours came from a local retailer), chosen solely because it was the cheapest 8 inch tablet available!

The tablet seems pretty good overall, at least for the price. The build quality is fine, although the case is very plastic-y. It’s a little slow after boot up as it loads up everything, but in normal usage as a wall mounted dashboard it performs perfectly well. The shipped operating system is Android 7.0, which is up to date enough to not present any problems with app support.

The biggest issue with this device is that it charges really slowly, even from the provided charger. I even had cases when setting it up that the battery would still drain when plugged in to the charger. I was a little concerned that the tablet battery would become discharged over time during normal use, due to wake ups from the motion sensing. However, once the tablet was mounted on the wall and left overnight it charged to 100% and has remained there since.

Fixing the Brackets

I printed new brackets for the tablet with a reduced front lip. This removes the issue with the brackets obscuring the screen. The right bracket (left from the point of view of the tablet and in the STL files) also has a cut out in it’s side to allow the USB cable to pass through. I made these edits in OpenSCAD, by first importing the original STL files. I then created an object which represents the material I wanted to remove. Then a difference operation between the two gives the final object. I’m pretty pleased with them. The only thing I wasn’t able to reproduce was the rounded internal edges on the cut out sections.

Wall Panel MQTT Integration

With the power issues solved, I can integrate the tablet fully with Home Assistant (without fear that the extra load will cause it to power off). I’ve integrated in the sensors available on this device (pretty much just the power status). The full YAML code for this is available on GitLab. This pretty much went according to the WallPanel documentation. However, I did find that I needed to specify a value_template in order to extract the battery value:

sensor:
  - platform: mqtt
    state_topic: "wallpanel/kitchen/sensor/battery"
    name: "Kitchen WallPanel Battery Level"
    unit_of_measurement: "%"
    value_template: '{{value_json.value}}'
    device_class: battery

To extract the boolean status for the charging state sensors, the template gets a little more complex:

binary_sensor:
  - platform: mqtt
    state_topic: "wallpanel/kitchen/sensor/battery"
    name: "Kitchen WallPanel Charge State"
    value_template: '{% if value_json.charging %}ON{% else %}OFF{% endif %}'
    device_class: power

  - platform: mqtt
    state_topic: "wallpanel/kitchen/sensor/battery"
    name: "Kitchen WallPanel AC State"
    value_template: '{% if value_json.acPlugged %}ON{% else %}OFF{% endif %}'
    device_class: plug

I also integrated in the motion, face detection and QR code sensors, although I’m not using the face detection or QR code sensors and have since disabled them in WallPanel.

sensor:
  - platform: mqtt
    state_topic: "wallpanel/kitchen/sensor/qrcode"
    name: "Kitchen QR Code"
    value_template: '{{value_json.value}}'

binary_sensor:
  - platform: mqtt
    state_topic: "wallpanel/kitchen/sensor/motion"
    name: "Kitchen Motion"
    payload_on: '{"value":true}'
    payload_off: '{"value":false}'
    device_class: motion

  - platform: mqtt
    state_topic: "wallpanel/kitchen/sensor/face"
    name: "Kitchen Face Detected"
    payload_on: '{"value":true}'
    payload_off: '{"value":false}'
    device_class: motion

The final integration I did was to add a switch to turn on and off the camera streaming. However, in general usage this seems a bit flakey, with the camera stream not usually becoming available until after a WallPanel restart. Here’s the YAML used for the switch anyway:

switch:
  - platform: mqtt
    command_topic: "wallpanel/kitchen/command"
    name: "Kitchen Camera State"
    payload_on: '{"camera": true}'
    payload_off: '{"camera": false}'
    optimistic: true
    retain: true

Dashboard Improvements

I’ve improved my dashboard somewhat since my last post. However, it’s still a long way from some of the awesome dashboards I’ve seen posted. It’s still a work in progress!

The main changes have been to switch to the animated weather custom card for the weather forecast and to add Custom Header to my install to compact the header. I then added more buttons, some media controls and a gauge showing the progress of my 3D prints. The vacuum buttons call some custom scripts to start the vacuum with our preferred settings. This is necessary to work around a bug in HASS with the Botvac D3:

script:
  spot_clean:
    alias: "Spot Clean"
    sequence:
      - alias: "Start spot clean"
        service: vacuum.neato_custom_cleaning
        data:
          entity_id: vacuum.marvin
          mode: 2
          category: 3

  house_clean:
    alias: "House Clean"
    sequence:
      - alias: "Start house clean"
        service: vacuum.neato_custom_cleaning
        data:
          entity_id: vacuum.marvin
          mode: 1
          category: 2

It should be noted that I’m running a slightly older version of HASS at the moment. In newer versions the service name changed to neato.custom_cleaning. Also, using a category value of 3 for spot cleaning appears to be undocumented, but if you look at the library, that’s what it’s doing.

Conclusions

This setup is now working perfectly and all the problems with the previous iteration have been resolved. It’s unfortunate that this was done by throwing money at the problems, but sometimes things just don’t work out as we hope!

Overall, I’m really happy with the finished product. The dashboard will be continually developing as I come up with ideas and work out what’s useful and what isn’t. Feel free to share your ideas again via the feedback channels.