KISS high-availability with OpenBSD

Art by Michael J. Penick (mod. by Paul B.)
                                               ACME-sky
        __________
       / nsd tower\                                             (
      /____________\                                           (\) awk-ward
       |:_:_:_:_:_|                                             ))   plant
       |_:_,--.:_:|                       dig-bubble         (\//   )
       |:_:|__|_:_|  relayd-castle          _               ) ))   ((
    _  |_   _  :_:|   _   _   _            (_)             ((((   /)\`
   | |_| |_| |   _|  | |_| |_| |             o              \\)) (( (
    \_:_:_:_:/|_|_|_|\:_:_:_:_/             .                ((   ))))
     |_,-._:_:_:_:_:_:_:_.-,_|                                )) ((//
     |:|_|:_:_:,---,:_:_:|_|:|                               ,-.  )/
     |_:_:_:_,'puffy `,_:_:_:_|           _  o               ,;'))((
     |:_:_:_/  _ | _  \_:_:_:|          (_O                   ((  ))
_____|_:_:_|  (o)-(o)  |_:_:_|--'`-.     ,--. ksh under-water (((\'/
 ', ;|:_:_:| -( .-. )- |:_:_:| ', ; `--._\  /,---.~  goat     \`))
.  ` |_:_:_|   \`-'/   |_:_:_|.  ` .  `  /()\.__( ) .,-----'`-\(( sed-root
 ', ;|:_:_:|    `-'    |:_:_:| ', ; ', ; `--'|   \ ', ; ', ; ',')).,--
.  ` MJP ` .  ` .  ` .  ` . httpd-soil ` .    .  ` .  ` .  ` .  ` .  `
 ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ; ', ;

Table of contents
=================

KISS high-availability with OpenBSD
    My auto-failover requirements
    My HA solution
        Only OpenBSD base installation required
        Fairly cheap and geo-redundant
        Failover time and split-brain
        Failover support for multiple protocols
        Let's encrypt TLS certificates
        Monitoring
        Rex automation
    More HA

My auto-failover requirements

• Be OpenBSD-based (I prefer OpenBSD because of the cleanliness and good documentation) and rely on as few external packages as possible.
• Don't rely on the hottest and newest tech (don't want to migrate everything to a new and fancier technology next month already!).
• It should be reasonably cheap. I want to avoid paying a premium for floating IPs or fancy Elastic Load Balancers.
• It should be geo-redundant.
• It's fine if my sites aren't reachable for five or ten minutes every other month. Due to their static nature, I don't care if there's a split-brain scenario where some requests reach one server and other requests reach another server.
• Failover should work for both HTTP/HTTPS and Gemini protocols. My self-hosted MTAs and DNS servers should also be highly available.
• Let's Encrypt TLS certificates should always work (before and after a failover).
• Have good monitoring in place so I know when a failover was performed and when something went wrong with the failover.
• Don't configure everything manually. The configuration should be automated and reproducible.

My HA solution

Only OpenBSD base installation required

#!/bin/ksh

ZONES_DIR=/var/nsd/zones/master/
DEFAULT_MASTER=fishfinger.buetow.org
DEFAULT_STANDBY=blowfish.buetow.org

determine_master_and_standby () {
    local master=$DEFAULT_MASTER
    local standby=$DEFAULT_STANDBY

    .
    .
    .
    
    local -i health_ok=1
    if ! ftp -4 -o - https://$master/index.txt | grep -q "Welcome to $master"; then
        echo "https://$master/index.txt IPv4 health check failed"
        health_ok=0
    elif ! ftp -6 -o - https://$master/index.txt | grep -q "Welcome to $master"; then
        echo "https://$master/index.txt IPv6 health check failed"
        health_ok=0
    fi
    if [ $health_ok -eq 0 ]; then
        local tmp=$master
        master=$standby
        standby=$tmp
    fi

    .
    .
    .
}

fishfinger$ grep failover /var/nsd/zones/master/foo.zone.zone
        300 IN A 46.23.94.99 ; Enable failover
        300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover
www     300 IN A 46.23.94.99 ; Enable failover
www     300 IN AAAA 2a03:6000:6f67:624::99 ; Enable failover
standby  300 IN A 23.88.35.144 ; Enable failover
standby  300 IN AAAA 2a01:4f8:c17:20f1::42 ; Enable failover

transform () {
  sed -E '
	/IN A .*; Enable failover/ {
	    /^standby/! {
	        s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/master_a)' ; \3/;
	    }
	    /^standby/ {
	        s/^(.*) 300 IN A (.*) ; (.*)/\1 300 IN A '$(cat /var/nsd/run/standby_a)' ; \3/;
	    }
	}
	/IN AAAA .*; Enable failover/ {
	    /^standby/! {
	        s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/master_aaaa)' ; \3/;
	    }
	    /^standby/ {
	        s/^(.*) 300 IN AAAA (.*) ; (.*)/\1 300 IN AAAA '$(cat /var/nsd/run/standby_aaaa)' ; \3/;
	    }
	}
	/ ; serial/ {
	    s/^( +) ([0-9]+) .*; (.*)/\1 '$(date +%s)' ; \3/;
	}
  '
}

#! Race condition !#
   
if [ -f $zone_file.bak ]; then
    mv $zone_file.bak $zone_file
fi

cat $zone_file | transform > $zone_file.new.tmp 

grep -v ' ; serial' $zone_file.new.tmp > $zone_file.new.noserial.tmp
grep -v ' ; serial' $zone_file > $zone_file.old.noserial.tmp

echo "Has zone $zone_file changed?"
if diff -u $zone_file.old.noserial.tmp $zone_file.new.noserial.tmp; then
    echo "The zone $zone_file hasn't changed"
    rm $zone_file.*.tmp
    return 0
fi

cp $zone_file $zone_file.bak
mv $zone_file.new.tmp $zone_file
rm $zone_file.*.tmp
echo "Reloading nsd"
nsd-control reload

if ! zone_is_ok $zone; then
    echo "Rolling back $zone_file changes"
    cp $zone_file $zone_file.invalid
    mv $zone_file.bak $zone_file
    echo "Reloading nsd"
    nsd-control reload
    zone_is_ok $zone
    return 3
fi

for cleanup in invalid bak; do
    if [ -f $zone_file.$cleanup ]; then
        rm $zone_file.$cleanup
    fi
done

echo "Failover of zone $zone to $MASTER completed"
return 1

Fairly cheap and geo-redundant

Failover time and split-brain

Failover support for multiple protocols

Let's encrypt TLS certificates

• In an odd week number, the first server is the default master
• In an even week number, the second server is the default master.

# Weekly auto-failover for Let's Encrypt automation
local -i -r week_of_the_year=$(date +%U)
if [ $(( week_of_the_year % 2 )) -eq 0 ]; then
    local tmp=$master
    master=$standby
    standby=$tmp
fi

Monitoring

Rex automation

More HA

Generated by Gemtexter 2.2.0-develop | served by OpenBSD/httpd(8) | Site Mirrors