docker common containers

Quick Project: Splitting Docker Compose Projects

This post may contain affiliate links. Please see the disclaimer for more information.

Way back in the when I first started using Docker in earnest, I wrote about my web hosting stack. Recently, this has undergone an upgrade as I’m working on a new website which will be served from the same server. I took the opportunity to split the system up into multiple docker-compose projects, which makes deployment of further sites much easier. It allows me to manage the common containers from one docker-compose project and then each of the sites from their own project. This will be of further use in future as I move towards deploying these with Ansible.

The Approach

My basic approach here is to move my two common containers (my Traefik container and SMTP forwarder) into their own project. This project will create a couple of networks for interfacing to the containers from other projects. To create these networks I add the following to my common project docker-compose.yml:

networks:
  gateway:
    name: 'gateway'
  smtp:
    name: 'smtp'

Here I create two networks as per normal. The key is to give them a proper name, rather than the auto generated one that would be assigned by Docker. This will enable us to address them easily from our other projects. We then assign these to our common containers:

services:
  traefik:
    image: traefik:2.1
    command:
      ...
    volumes:
      ...
    ports:
      - "80:80"
      - "443:443"
      - "127.0.0.1:8080:8080"
    networks:
      gateway:
        aliases:
          # add hostnames you might want to refer to this container by
          - example.com
    restart: always

  postfix:
    image: boky/postfix
    ports:
      ...
    environment:
      ...
    volumes:
      ...
    networks:
      smtp:
        aliases:
          - postfix
    restart: always

Here I simply assign the relevant network to each container. The aliases section allows other containers on these networks to find our common containers by whatever name we specify. In the case of the postfix container this is to connect via SMTP. For the traefik container, adding hostnames which internal apps my need to refer to can help (for example with the WordPress loopback test).

External Projects

With this in place, the other applications can be moved out into their own projects. For each one we need to access the gateway and smtp networks in order to have access to our common services. These are accessed as external networks via the docker-compose.yaml file for our project:

networks:
  gateway:
    external: true
  smtp:
    external: true

We then go ahead and add our services to access these networks:

services:
  varnish:
    image: wodby/varnish:latest
    depends_on:
      - wordpress
    environment:
      ...
    labels:
      - 'traefik.enable=true'
      - "traefik.docker.network=gateway"
      ...
    networks:
      - gateway
      - cache
    restart: always

  wordpress:
    image: wordpress:latest
    depends_on:
      - mariadb
    environment:
      ...
    volumes:
      ...
    networks:
      smtp:
      cache:
      database:
    restart: always

Here I add my varnish cache, as per my previous article. The key thing here is to specify the label traefik.docker.network=gateway to allow Traefik to reliably discover the container. We then also make sure the container is added to the gateway network. I’ve also added a WordPress container, which is on the smtp network. This will allow sending of email from WordPress via the SMTP forwarder.

Conclusion

This is a pretty simple approach for better management of my increasingly complex web stack. As I mentioned earlier that the next step will be to deploy these projects via Ansible. In this case the common containers will become part of a role which can be used across my infrastructure.

The splitting out of the apps into their own projects has enabled me to duplicate my current WordPress+Varnish+Mariadb setup for the new site I’m working on. There will be more info to come about that site as soon as I am ready to share!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

rss-bridge

Reconnecting the Web with RSS-Bridge

This post may contain affiliate links. Please see the disclaimer for more information.

I’ve mentioned before that I’m a big fan of RSS as a medium for consuming my daily news and for following the blogs of others. However, there are an increasing number of websites that don’t provide an RSS feed (or at least don’t advertise a feed if one exists). Luckily for us there is an awesome piece of self-hosted software which aims to fill in the gaps left by these missing feeds – RSS-bridge.

My use case for this was twofold. First I wanted to follow some sites for which I couldn’t find RSS feeds, specifically The Guardian. Second, I wanted to get updates from some local groups, who only have a Facebook page. Obviously, I don’t actually want to actually check in to Facebook to do this, that would be intolerable. RSS-Bridge fills both these needs.

Installation

There are several public instances of RSS-bridge available, but of course I wanted to host my own. Doing so is extremely easy with Docker. I added the following to my docker-compose.yml file on the server in question:

services:
  rss-bridge:
    image: rssbridge/rss-bridge:latest
    volumes:
      - /mnt/docker-data/rss-bridge/whitelist.txt:/app/whitelist.txt
    labels:
      - 'traefik.enable=true'
      - "traefik.http.middlewares.rssbridge_redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.rssbridge_insecure.rule=Host(`rssbridge.example.com`)"
      - "traefik.http.routers.rssbridge_insecure.entrypoints=web"
      - "traefik.http.routers.rssbridge_insecure.middlewares=rssbridge_redirect@docker"
      - "traefik.http.routers.rssbridge.rule=Host(`rssbridge.example.com`)"
      - "traefik.http.routers.rssbridge.entrypoints=websecure"
      - "traefik.http.routers.rssbridge.tls.certresolver=mydnschallenge"
      - "traefik.http.services.rssbridge.loadbalancer.server.port=80"
    networks:
      - external
    restart: always

This uses Traefik, with my internal HTTPS setup to serve the bridge over HTTPS. You can also set up authentication for the bridge if you like. This isn’t really required unless you are hosting the bridge on a publicly available URL and would rather keep it private. I elected not to bother with authentication, since mine is on my internal network. It should also be noted that the bridge is totally stateless. All the parameters are sent in the URL, so there is no data to protect.

Grabbing Feeds

You’ll see above that we mounted a text file called whitelist.txt inside the container. This contains a list of all the bridges you want to use, from the full list of bridges. Here’s mine:

FacebookBridge
TheGuardianBridge
TwitterBridge
YoutubeBridge

I’ll demonstrate the use of a couple of these below, but it’s pretty simple. First up TheGuardianBridge, just select the section of the site you are interested in and click a button – couldn’t be easier!

rss-bridge
Super simple!

I like to use the HTML button so that I can see that the bridge is working right there in the browser. You can then grab the (M)RSS or Atom links directly from the resulting page:

rss-bridge
The resulting feed page

I’m also going to grab a feed of my local council news from their Facebook page, using the FacebookBridge:

rss-bridge
The Facebook Bridge

Here we just enter the name of the page or user we are interested in. There is another dialogue below this for groups, but I haven’t tried that yet. I assume this only works for public pages, since it doesn’t ask for any login credentials. Of course, when we click through we are greeted by our feed:

rss-bridge
The resultant Facebook feed

The Twitter bridge works similarly. I haven’t had much luck with the Youtube bridge, but I’m already using a well known trick to get RSS feeds of my favourite Youtube channels.

Setting Up Email Notifications

So far, this has all been very easy. Let’s step it up (just a little) and get notified when one of our feeds gets updated. I’m using this to be notified of events and goings on in my local area via some of the Facebook feeds. This closes the loop quite nicely and takes “social media” back to the promise it had in the early days.

To do this I’m using a tool called rss2email. This is a brilliant little tool, which I actually used as my primary RSS reader for some years, until I got too many feeds to get through all the emails! I’m glad to press it back into service for this.

I elected not to install rss2email in Docker, since I couldn’t find a nicely updated image and didn’t fancy building my own. It’s also kind of a personal tool, so fits nicely in a Unix user account as a cron job. On Ubuntu rss2email can be installed via APT:

$ sudo apt install rss2email

Next it’s best to follow the official documentation to get it up and running. You’ll need some access to an SMTP server to be able to send mail. One place where the documentation seems to differ is in enabling SMTP, where I had to use the line email-protocol = smtp rather than the use-smtp specified in the docs.

Once this is all set up you can add your feeds like so:

$ r2e add FeedName https://rss-bridge.example.com/.....

Of course you can add non-RSS-bridge feeds too. Just add whatever feeds you’d like to receive notifications on!

The last thing is to schedule this as a cron job:

14  *  *   *   *     /usr/local/bin/log-output "/usr/bin/r2e run"

I’m using the wrapper script I’ve mentioned previously. Done!

Conclusion

This has been a really simple project (by my standards). Everything went according to plan, which almost never happens! Regardless, I’m very happy with the result and it’s something I’ll continue to make use of every day.

RSS-Bridge fills a much needed hole in the modern web. With the dominance of the big social media platforms and increasing “appification”, we’ve lost the real promise of the web to be an open and connected platform. RSS-bridge brings back at least some of this.

The addition of rss2email fulfils the hopes I had for social networks in the early days – that they would become notification platforms for events/people/things in the world around us. Instead, they’ve become locked down walled gardens which force you to use their app or website in order to engage with what’s going on.

Technology should come to us, on our own terms and via whatever medium we choose. This makes projects like RSS-Bridge, rss2email and the myriad of RSS readers out there incredibly important for those who refuse to be locked inside the gardens, but still require access to the information contained within.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

restic rsync backups

Centralised Backups With Restic and Rsync

This post may contain affiliate links. Please see the disclaimer for more information.

In my recent post on synchronising ZFS snapshots from remote servers, I mentioned that I had being using rsync for the same purpose. This is part of my larger overall backup strategy with restic. It was brought to my attention recently that I hadn’t actually written up my backup approach. This post will rectify that!

The key requirement of my system was to have something that would work across multiple systems, without being too difficult to maintain. I also wanted it to scale to new systems easily as my self-hosting infrastructure inevitably continues to grow. Of course, I had the usual requirements of local and off site backups, with the off site copy suitably secured. Restic fits the bill quite nicely for secure local and remote backups, but has no way to synchronise multiple systems unless you set it up on each system individually.

Backup Architecture

I’ve architected my backups as a centralised system, where the relevant data from each satellite system is propagated to a central server and then backed up to various end points from there. This architecture was chosen because it was reasonably easy for me to setup and maintain and actually results in more copies of the data since it has to be copied to the backup server first.

restic rsync backups
They say a picture is worth a thousand words…

As you can see from the diagram the synchronisation from the remote systems to the backup server is done via rsync. This is done in a pull fashion. The backup server connects to each machine in turn and pulls down the files to be backed up to it’s local cache.

The second stage is a backup using restic to both a locally connected external hard drive and to the cloud (in this case Backblaze B2). I’ll cover each of these steps in the following sections.

Synchronising Remote Machines with Rsync

The first step is to synchronise the relevant files on the remote machines via rsync. When I say remote machines here, I specifically mean machines which are not the central backup server. These could be remote cloud machines, hosts on the local network or VMs hosted on the same machine. In my case it’s all three, since I run the backups on my main home server.

For each machine I want to synchronise, I have a script looking like this:

#!/bin/bash

HOST=<REMOTE HOSTNAME>
PORT=22
USER=backup
SSH_KEY="/storage/data/backup/keys/backup_key"

BASE_DEST=/storage/data/backup/$HOST

LOG_DIR=/storage/data/backup/logs
LOG_FILE=$LOG_DIR/rsync-$HOST.log

function do_rsync() {
    echo "Starting rsync job for $HOST:$1 at $(date '+%Y-%m-%d %H:%M:%S')..." >> $LOG_FILE 2>&1
    echo >> $LOG_FILE 2>&1

    mkdir -p $BASE_DEST$1 >> $LOG_FILE 2>&1
    /usr/bin/rsync -avP --delete -e "ssh -p $PORT -i $SSH_KEY" $USER@$HOST:$1 $BASE_DEST$1 >> $LOG_FILE 2>&1

    echo >> $LOG_FILE 2>&1
    echo "Job finished at $(date '+%Y-%m-%d %H:%M:%S')." >> $LOG_FILE 2>&1
}

mkdir -p $LOG_DIR

do_rsync <DIRECTORY 1>
do_rsync <DIRECTORY 2>
...

Here we start with some basic configuration, including the hostname, port, user and SSH key to use to connect to the remote host. I then configure the local destination directory, which is located on my main ZFS mirror. I also configure where the logs will be stored.

We then get into the main function of the script, called do_rsync. This sets up the logging environment and does the actual rsync transfer with the options we’ve specified. It takes as an argument the remote directory to backup (which obviously must be readable by the user in question).

We then close out the script by ensuring the log directory exists and then calling the do_rsync function for the directories we are interested in. Looking at the backup scripts now it would actually be good to factor out the common functionality here into a helper script. This could then be sourced by all of the host specific scripts. I also need to move this into git which will happen with my continued migration to Ansible.

A Note About Security

Obviously, with the rsync client logging in to the remote system automatically via SSH it’s good to restrict what this can do. To this end, the SSH key is locked down so that the only command that can be run is that executed by the rsync client. This is done via the ~/.ssh/authorized_keys file:

command="/usr/bin/rsync --server --sender -vlogDtpre.is . ${SSH_ORIGINAL_COMMAND//* \//\/}",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-rsa .....

These backup scripts are run from cron and spread throughout the day so as not to overlap with each other, in an effort to even out the traffic on the network. I’m not really happy with this part of the solution, it might just be better to run the whole lot in sequence.

Local Restic Backups

The next step in the process is to run the backup to the locally connected external drive with restic. This backup is run over all the previously synced data as well as data from the local machine, such as the contents of my Nextcloud server and media collection.

This backup is achieved with the following script:

#!/bin/bash
set -e

######## START CONFIG ########
DISK_UUID="<DISK UUID>"
#GLOBAL_FLAGS="-q"
MOUNT_POINT=/mnt/backup
export RESTIC_REPOSITORY=$MOUNT_POINT/restic/storage
export RESTIC_PASSWORD_FILE=/root/restic-local.pass
######## END CONFIG ########

echo "Starting backup process at $(date '+%Y-%m-%d %H:%M:%S')."

# check for the backup disk and mount it
if [ ! -e /dev/disk/by-uuid/$DISK_UUID ]; then
    echo "Backup disk not found!" >&2
    exit 1
fi
echo "Mounting backup disk..."
mount -t ext4 /dev/disk/by-uuid/$DISK_UUID $MOUNT_POINT

# pre-backup check
echo "Performing pre-backup check..."
restic $GLOBAL_FLAGS check

# perform backups
echo "Performing backups..."
restic $GLOBAL_FLAGS backup /storage/data/nextcloud
restic $GLOBAL_FLAGS backup /storage/data/backup
restic $GLOBAL_FLAGS backup /storage/music
restic $GLOBAL_FLAGS backup /storage/media
# add any other directories here...

# post-backup check
echo "Performing post backup check..."
restic $GLOBAL_FLAGS check

# clean up old snapshots
echo "Cleaning up old snapshots..."
restic $GLOBAL_FLAGS forget -d 7 -w 4 -m 6 -y 2 --prune

# final check
echo "Performing final backup check..."
restic $GLOBAL_FLAGS check

# unmount backup disk
echo "Unmounting backup disk..."
umount $MOUNT_POINT

echo "Backups completed at $(date '+%Y-%m-%d %H:%M:%S')."
exit 0

This script is pretty simple, despite the wall of commands. First we have some configuration in which I specify the UUID of the external disk and the mount point of the disk. This is done because the disk is kept unmounted when not in use. The path to the restic repository, relative to the mount point and the path to the password file are also specified.

We then move into checking for and mounting the external disk. The first restic command performs a check on the repository to make sure all is well, before getting into the backups for the directories we are interested in. This is followed by another check to make sure that went OK.

I then run a restic forget command to prune old snapshots. Currently I’m keeping the last 7 days of backups, 4 weekly backups, 6 monthly backups and 2 yearly backups! I run a final restic check before unmounting the external disk.

This script is run once a day from cron. I use the following command to reduce the priority of the backup script and avoid interfering with the normal operation of the server:

/usr/bin/nice -n 19 /usr/bin/ionice -c2 -n7 /storage/data/backup/restic-local.sh >> /storage/data/backup/logs/restic-local.log

Remote Restic Backups

The final stage in this process is a separate backup to the cloud. As mentioned before I use Backblaze’s B2 service for this since it seems to be about the cheapest around. I’ve been reasonably happy with it so far at least.

#!/bin/bash
set -e

######## START CONFIG ########
B2_CONFIG=/root/b2_creds.sh
#GLOBAL_FLAGS="-q"
export RESTIC_REPOSITORY="<MY B2 REPO>"
export RESTIC_PASSWORD_FILE=/root/restic-remote-b2.pass
######## END CONFIG ########

echo "Starting backup process at $(date '+%Y-%m-%d %H:%M:%S')."

# load b2 config
source $B2_CONFIG

# pre-backup check
echo "Performing pre-backup check..."
restic $GLOBAL_FLAGS check

# perform backups
echo "Performing backups..."
restic $GLOBAL_FLAGS backup /storage/data/nextcloud
restic $GLOBAL_FLAGS backup /storage/data/backup
restic $GLOBAL_FLAGS backup /storage/music
# This costs to much to backup, but it's not the end of the world
# if I lose a load of DVD rips!
#restic $GLOBAL_FLAGS backup /storage/media

# post-backup check
echo "Performing post backup check..."
restic $GLOBAL_FLAGS check

# clean up old snapshots
echo "Cleaning up old snapshots..."
restic $GLOBAL_FLAGS forget -w 8 -m 12 -y 2 --prune

# final check
echo "Performing final backup check..."
restic $GLOBAL_FLAGS check

echo "Backups completed at $(date '+%Y-%m-%d %H:%M:%S')."
exit 0

This looks very similar to the previous script, but differs in the configuration. First I specify the file where I keep my B2 credentials, to be sourced later. This file is of the form:

export B2_ACCOUNT_ID="<MY ACCOUNT ID>"
export B2_ACCOUNT_KEY="<MY ACCOUNT KEY>"

I then set the RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE variables as before. In this case the repository is of the form b2:bucketname:path/to/repo.

The snapshot retention policy here is different, with 8 weekly backups, 12 monthly backups and 2 yearly backups retained. This is mostly because I only run this backup once per week – the backup script will actually take more than 24 hours to run with all the checking and forgetting thrown in! The script is run from cron with the same nice/ionice combination as the local backup.

Conclusion

With all that in place, I have a pretty comprehensive backup system. The system stores at least 3 copies of any data (live, local backup, remote backup) and in the case of remote systems 4 copies (live, backup server cache, local backup, remote backup). The main issue I have with this setup currently is the use of the local external disk, which I don’t like being connected to the same server. Hopefully I’ll be moving this to another machine in my next round of server upgrades.

I also don’t really like the reliance on the cloud, even though I’ve got no complaints about the B2 service. My ideal system would probably be an SBC based system located at the home of someone with a fast, non-data capped internet connection. Ideally this person would also live on a different continent! I could then run a Minio server in place of the B2 service. This would probably end up cheaper in the long run, since I’m paying nearly $10/month for the current service.

One last piece of sage advice: TEST YOUR BACKUPS! They are worth nothing if you don’t know they are working. I’ve done a couple of test restores with this system, but I’m probably due for another one.

What’s your backup routine like? Got improvements for my system? Feel free to share them in the comments.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

Integrating Remote Servers Into My Local Network

This post may contain affiliate links. Please see the disclaimer for more information.

I’ve been using Linode for many years to host what I consider to be my most “production grade” self-hosted services, namely this blog and my mail server. My initial Linode server was built in 2011 on CentOS 6. This is approaching end of life so, I’ve been starting to build its replacement. Since originally building this server my home network has grown up and now provides a myriad of services. When starting out to build the new server, I thought it would be nice to be able to make use of these more easily from my remote servers. So I’ve begun some work to integrate the two networks more closely.

Integration Points

There are a few integration points I’m targeting here, some of which I’ve done already and others are still to be done:

  • Get everything onto the same network, or at least on different subnets of my main network so I can control traffic between networks via my pfSense firewall. This gives me the major benefit of being able to access selected services on my local network from the cloud without having to make that service externally accessible. In order to do this securely you want to make sure the connection is encrypted – i.e. you want a VPN. I’m using OpenVPN.
  • Use ZFS snapshots for backing up the data on the remote systems. I’d previously been using plain old rsync for copying the data down locally where it gets rolled into my main backups using restic. There is nothing wrong with this approach, but using ZFS snapshots gives more flexibility for restoring back to a certain point without having to extract the whole backup.
  • Integrate into my check_mk setup for monitoring. I’m already doing this and deploying the agent via Ansible/CI. However, now the agent connection will go via the VPN tunnel (it’s SSH anyway, so this doesn’t make a huge difference).
  • Deploy the configuration to everything with Ansible and Gitlab CI – I’m still working on this!
  • Build a centralised logging server and send all my logs to it. This will be a big win, but sits squarely in the to-do column. However, it will benefit from the presence of the VPN connection, since the syslog protocol isn’t really suitable for running over the big-bad Internet.

Setting Up OpenVPN

I’m setting this up with the server being my local pfSense firewall and the clients being the remote cloud machines. This is somewhat the reverse to what you’d expect, since the remote machines have static IPs. My local IP is dynamic, but DuckDNS does a great job of not making this a problem.

The server setup is simplified somewhat due to using pfSense with the OpenVPN Client Export package. I’m not going to run through the full server setup here – the official documentation does a much better job. One thing worth noting is that I set this up as the second OpenVPN server running on my pfSense box. This is important as I want these clients to be on a different IP range, so that I can firewall them well. The first VPN server runs my remote access VPN which has unrestricted access, just as if I were present on my LAN.

In order to create the second server, I just had to select a different UDP port and set the IP range I wanted in the wizard. It should also be noted that the VPN configuration is set up not to route any traffic through it by default. This prevents all the traffic from the remote server trying to go via my local network.

On the client side, I’m using the standard OpenVPN package from the Ubuntu repositories:

$ sudo apt install openvpn

After that you can extract the configuration zip file from the server and test with OpenVPN in your terminal:

$ unzip <your_config>.zip
$ cd <your_config>
$ sudo openvpn --config <your_config>.ovpn

After a few seconds you should see the client connect and should be able to ping the VPN address of the remote server from your local network.

Always On VPN Connection

To make this configuration persistent we first move the files into /etc/openvpn/client, renaming the config file to give it the .conf extension:

$ sudo mv <your_config>.key /etc/openvpn/client.key
$ sudo mv <your_config>.p12 /etc/openvpn/client.p12
$ sudo mv <your_config>.ovpn /etc/openvpn/client.conf

You’ll want to update the pkcs12 and tls-auth lines to point to the new .p12 and .key files. I used full paths here just to makes sure it would work later. I also added a route to my local network in the client config:

route 10.0.0.0 255.255.0.0

You should then be able to activate the OpenVPN client service via systemctl:

$ sudo systemctl start openvpn-client@client.service
$ sudo systemctl enable openvpn-client@client.service

If you check your system logs, you should see the connection come up again. It’ll now persist across reboots and should also reconnect if the connection goes down for any reason. So far it’s been 100% stable for me.

At this point I added a DNS entry on my pfSense box to allow me to access the remote machine via it’s hostname from my local network. This isn’t required, but it’s quite nice to have. The entry points to the VPN address of the machine, so all traffic will go via the tunnel.

Firewall Configuration

Since these servers have publicly available services running on them, I don’t want them to have unrestricted access to my local network. Therefore, I’m blocking all incoming traffic from the new VPN’s IP range in pfSense. I’ll then add specific exceptions for the services I want them to access. This is pretty much how you would set up a standard DMZ.

remote server integrate
The firewall rules for the OpenVPN interface, note the SSH rule to allow traffic for our ZFS snapshot sync later

To do this I added an alias for the IP range in question and then added a block rule on the OpenVPN firewall tab in pfSense. This is slightly different to the way my DMZ is set up, since I don’t want to block all traffic on the OpenVPN interface, just traffic from that specific IP range (to allow my remote access VPN to continue working!).

You’ll probably also want to configure the remote server to accept traffic from the VPN so that you can access any services on the server from your local network. Do this with whatever Linux firewall tool you usually use (I use ufw).

Storing Data on ZFS

And now for something completely different….

As discussed before, I was previously backing up the data on these servers with rsync. However, I was missing the snapshotting I get on my local systems. These local systems mount their data directories via NFS to my main home server, which then takes care of the snapshot duties. I didn’t want to use NFS over the VPN connection for performance reasons, so I opted for local snapshots and ZFS replication.

In order to mount a ZFS pool on our cloud VM we need a device to store our data on. I could add some block storage to my Linodes (and I may in future), but I can also use a loopback file in ZFS (and not have to pay for extra space). To do this I just created a 15G blank file and created the zpool on top of that:

$ sudo mkdir /zpool
$ sudo dd if=/dev/zero of=/zpool/storage bs=1G count=15
$ sudo apt install zfsutils-linux
$ sudo zpool -m /storage storage /zpool/storage

I can then go about creating my datasets (one for the mail storage and one for docker volumes):

sudo zfs create storage/mail
sudo zfs create storage/docker-data

Automating ZFS Snapshots

To automate my snapshots I’m using Sanoid. To install it (and it’s companion tool Syncoid) I did the following:

$ sudo apt install pv lzop mbuffer libconfig-inifiles-perl libconfig-inifiles-perl git
$ git clone https://github.com/jimsalterjrs/sanoid
$ sudo mv sanoid /opt/
$ sudo chown -R root:root /opt/sanoid
$ sudo ln /opt/sanoid/sanoid /usr/sbin/
$ sudo ln /opt/sanoid/syncoid /usr/sbin/

Basically all we do here is install a few dependencies and then download Sanoid and install it in /opt. I then hard link the sanoid and syncoid executables into /usr/sbin so that they are on the path. We then need to copy over the default configuration:

$ sudo mkdir /etc/sanoid
$ sudo cp /opt/sanoid/sanoid.conf /etc/sanoid/sanoid.conf
$ sudo cp /opt/sanoid/sanoid.defaults.conf /etc/sanoid/sanoid.defaults.conf

I then edited the sanoid.conf file for my setup. My full configuration is shown below:

[storage/mail]
        use_template=production

[storage/docker-data]
        use_template=production
        recursive=yes

#############################
# templates below this line #
#############################

[template_production]
        frequently = 0
        hourly = 36
        daily = 30
        monthly = 12
        yearly = 2
        autosnap = yes
        autoprune = yes

This is pretty self explanatory. Right now I’m keeping loads of snapshots, I’ll pare this down later if I start to run out of disk space. The storage/docker-data dataset has recursive snapshots enabled because I will most likely make each Docker volume its own dataset.

This is all capped off with a cron job in /etc/cron.d/zfs-snapshots:

*  *    * * *   root    TZ=UTC /usr/local/bin/log-output '/usr/sbin/sanoid --cron'

Since my rant a couple of weeks ago, I’ve been trying to assemble some better practices around cron jobs. The log-output script is one of these, from this excellent article.

Syncing the Snapshots Locally

The final part of the puzzle is using Sanoid’s companion tool Syncoid to sync these down to my local machine. This seems difficult to do in a secure way, due to the permissions that zfs receive needs. I tried to use delegated permissions, but it looks like the mount permission doesn’t work on Linux.

The best I could come up with was to add a new unprivileged user and allow it to only run the zfs command with sudo by adding the following via visudo:

syncoid ALL=(ALL) NOPASSWD:/sbin/zfs

I also set up an SSH key on the remote machine and added it to the syncoid user on my home server. Usually I would restrict the commands that could be run via this key for added security, but it looks like Syncoid does quite a bit so I wasn’t sure how to go about this (if any one has any idea let me know).

With that in place we can test our synchronisation:

$ sudo syncoid -r storage/mail syncoid@<MY HOME SERVER>:storage/backup/mail
$ syncoid -r storage/docker-data syncoid@<MY HOME SERVER>:storage/docker/`hostname`

For this to work you should make sure that the parent datasets are created on the receiving server, but not the destination datasets themselves, Syncoid likes to create them for you.

I then wrote a quick script to automate this which I dropped in /root/replicator.sh:

#!/bin/bash

USER=syncoid
HOST=<MY HOME SERVER>

HOSTNAME=$(hostname)

/usr/sbin/syncoid -r storage/mail $USER@$HOST:storage/backup/mail 2>&1
/usr/sbin/syncoid -r storage/docker-data $USER@$HOST:storage/docker/$HOSTNAME 2>&1

Then another cron job in /etc/cron.d/zfs-snapshots finishes the job:

56 *    * * *   root    /usr/local/bin/log-output '/root/replicator.sh'

Conclusion

Phew! There was quite a bit there. Thanks for sticking with me if you made it this far!

With this setup I’ve come a pretty long way towards my goal of better integrating my remote servers. So far I’ve only deployed this to a single server (which will become the new mailserver). There are a couple of others to go, so the next step will be to automate as much as possible of this via Ansible roles.

I hope you’ve enjoyed this journey with me. I’m interested to hear how others are integrating remote and local networks together. Let me know if you have anything to add via the feedback channels.

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.

tiny tiny rss docker

Self-Hosted RSS with Tiny Tiny RSS in Docker

This post may contain affiliate links. Please see the disclaimer for more information.

With the rise of social media, RSS seems to have been largely forgotten. However, there are still those who are dedicated enough to keep curating their own list of feeds and plenty of software to support them. I’ve always been a fan of RSS and believe it’s probably time for a resurgence in use that would free us from our algorithmic overlords. As such I’ve run an instance of Tiny Tiny RSS for several years and the time has come to migrate it to Docker.

It’s relatively unknown that you can still get RSS feeds for most news sources on the Internet. For example, I use TT-RSS to keep up with Youtube and Reddit (just add .rss to the end of any subreddit URL) as well as the usual blogs and news sites.

About Tiny Tiny RSS

Tiny Tiny RSS is a web based RSS application (think Google Reader replacement). It’s PHP based and supports Postgres or MySQL (like) databases. I’ve been using it for may years and although I’ve tried out other web based RSS readers (such as Miniflux), I’ve never found anything as good as TT-RSS.

tiny tiny rss docker
My Tiny Tiny RSS install

A particular favourite feature of mine is the ability to generate feeds from any internal view, which makes it great for integrating with other systems which may consume RSS/Atom.

There is also an Android app which is available via the Play Store or F-Droid.

Finding a Tiny Tiny RSS Docker Image

I had initially planned to use the LinuxServer.io TT-RSS image, but it seems to have been deprecated. With a bit of searching I found this repo, however it’s pretty out of date and doesn’t build any more. After looking through my GitLab repos, it turned out I’d already tried upgrading that image as part of one of my previous attempts with Docker. I’ve finished off this migration and made the repo public so everyone can benefit from my efforts. Thanks to some CI magic and GitLab’s built in container registry you can pull the latest version like so:

docker pull registry.gitlab.com/robconnolly/docker-ttrss

Feel free to read through the project README to familiarise yourself with the options available in the image. It’s pretty much as it was in the original repository, so I’ll go through my setup below.

Setup with docker-compose

I’m integrating this with my existing Docker setup via docker-compose and Traefik. As such I added the following to my docker-compose.yml file:

ttrss:
    image: registry.gitlab.com/robconnolly/docker-ttrss:latest
    depends_on:
      - mariadb-ttrss
    environment:
      DB_NAME: ${TTRSS_DB_NAME}
      DB_USER: ${TTRSS_DB_USER}
      DB_PASS: ${TTRSS_DB_USER_PASSWD}
      DB_HOST: mariadb-ttrss
      DB_PORT: 3306
      DB_TYPE: mysql
      SELF_URL_PATH: https://ttrss.example.com
    volumes:
      - /mnt/docker-data/ttrss/plugins:/var/www/plugins.local
    labels:
      - 'traefik.enable=true'
      - "traefik.http.middlewares.ttrss_redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.ttrss_insecure.rule=Host(`ttrss.example.com`)"
      - "traefik.http.routers.ttrss_insecure.entrypoints=web"
      - "traefik.http.routers.ttrss_insecure.middlewares=ttrss_redirect@docker"
      - "traefik.http.routers.ttrss.rule=Host(`ttrss.example.com`)"
      - "traefik.http.routers.ttrss.entrypoints=websecure"
      - "traefik.http.routers.ttrss.tls.certresolver=mydnschallenge"
      - "traefik.http.services.ttrss.loadbalancer.server.port=80"
    networks:
      - external
      - internal
    restart: always

Breaking this down, we first create a new service using my TT-RSS image. We then define a dependency on the database container, which we will create later. The environment configuration uses another file env.sh in which we store our secrets. This is of the form:

export TTRSS_DB_ROOT_PASSWD=supersecret
export TTRSS_DB_USER=ttrss
export TTRSS_DB_USER_PASSWD=justalittlebitsecret
export TTRSS_DB_NAME=ttrss

In order to use this the file must be sourced before running docker-compose:

$ source env.sh
$ docker-compose .... # whatever you're doing

We can see that the database is configured entirely via environment variables as shown in the project README. We also set the SELF_URL_PATH variable so that TT-RSS knows where it is located (the URL should be updated for your configuration). I also chose to mount the plugins.local directory on the host machine to allow me to install plugins easily. The remainder of the configuration is for Traefik and is covered in my earlier post (you’ll need to update the hostnames used here too).

Database Setup

As mentioned earlier, we need a database container for TT-RSS to talk to. I’m using MariaDB for this because it’s what I’m familiar with. Also my original TT-RSS installation was in mysql and I wanted to migrate the data. The setup for this is pretty simple using the official MariaDB image:

mariadb-ttrss:
    image: mariadb
    environment:
      MYSQL_ROOT_PASSWORD: ${TTRSS_DB_ROOT_PASSWD}
      MYSQL_USER: ${TTRSS_DB_USER}
      MYSQL_PASSWORD: ${TTRSS_DB_USER_PASSWD}
      MYSQL_DATABASE: ${TTRSS_DB_NAME}
    volumes:
      - /mnt/docker-data/mariadb-ttrss/init/ttrss.sql.gz:/docker-entrypoint-initdb.d/backup.sql.gz
      - /mnt/docker-data/mariadb-ttrss/data:/var/lib/mysql
    networks:
      - internal
    restart: always

As you can see, I re-use the previous environment variables to create the database and user. I also mount the mysql data directory locally and mount a compressed backup of my previous database. This backup will only be loaded the first time the database comes up. You can remove this line if you are doing a clean install.

If you are following my install you’ll also see that I use a couple of Docker networks:

networks:
  external:
  internal:

The external network connects the local service containers to Traefik, whilst the internal network is used between TT-RSS and the database container.

With all this in place you should be able to launch your TT-RSS server with:

docker-compose up -d

At this point, it’s usually a good idea to check the container logs for problems and adjust your configuration accordingly.

Conclusion

Aside from having to update the Docker image for TT-RSS (which took quite a while) this migration was relatively painless. I’m quite happy with my newly Dockerised TT-RSS server. In addition to migrating it into Docker this step has also moved it off my ageing mailserver in preparation for it’s upcoming migration to something newer and moved it from the cloud onto my home server. All positive steps!

Next Steps

I’m pretty keen to keep maintaining my new Docker image for Tiny Tiny RSS, since it seems to be a gap in the community that can be filled. I’m currently building this in CI, but the configuration is pretty basic. In the coming weeks, I’m intending to expand upon this a little and set up a scheduled build which will automatically keep the container up to date. This will hopefully be the topic of a follow up article, so stay tuned!

If you liked this post and want to see more, please consider subscribing to the mailing list (below) or the RSS feed. You can also follow me on Twitter. If you want to show your appreciation, feel free to buy me a coffee.