Master-Master PowerDNS with Galera Replication

This walk-through will show you how to setup PowerDNS to serve as authoritative nameservers, using a Galera cluster to replicate records between the hosts. 

Using PowerDNS in this manner allows records and zones to be updated on the fly, without needing to reload or restart the service. Additionally, using Galera as a backend will allow for very easy edits via either SQL, one of many tools already out there, or even your own custom tooling. 

For this tutorial, we will need 3 servers total. It may be wise to separate these on different providers if you plan to use them in production, however I am just going to spin up a few Digital Ocean Droplets. You can check them out here: https://m.do.co/c/c18a5034b55e

Additionally, I will be using CentOS 7 as my operating system. PowerDNS and Galera support many other operating systems though, and the general process should carry over to anything else. File paths and commands may differ slightly, however. 

I’ve got my 3 servers:

pdns1.zswap.net - 104.248.9.38
pdns2.zswap.net - 104.248.9.33
pdns3.zswap.net - 104.248.9.37

First, login and make sure your system is up to date:

yum update -y

Next, we’ll setup the Galera cluster. Go to https://downloads.mariadb.org/mariadb/repositories/ and get the repo information for your OS of choice. In my case, it is:

# MariaDB 10.3 CentOS repository list - created 2018-10-18 23:19 UTC
# http://downloads.mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.3/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

Once you’ve added the repo to each of your servers, installed the required packages:

yum install MariaDB-server MariaDB-client socat

It should pull in Galera as a dependency.

Start MariaDB, and run the mysql_secure_installation script. Disable remote root logins, and set a password that you’ll remember. I recommend generating a random one as this will effectively be replicated to each node in the cluster, later on. 

Enable the MariaDB service on boot, but don’t start it just yet:

systemctl enable mariadb

On your first server, open up /etc/my.cnf.d/server and find the [mysqld] section.

Paste in the following lines into the configuration file:

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name="pdns_galera"
wsrep_cluster_address="gcomm://104.248.9.38,104.248.9.33,104.248.9.37"
wsrep_sst_method=rsync
wsrep_node_address="104.248.9.38"
wsrep_node_name="pdns1"

Be sure to change the wsrep_cluster_address, wsrep_node_address, and wsrep_node_name to reflect your server, and not mine. 

Note, if you are on a Debian based OS, the path for the library should be lib, not lib64. 

If your servers have working DNS for their hostname, you can use those. I prefer to use IP addresses as a fail-safe. Either will work. 

Once you’ve got the correct settings in there, close the file. 

Then bootstrap the cluster with:

galera_new_cluster

This is a bash script that handles setting up the initial cluster. Anytime you are starting from nothing (for example, if all nodes in the cluster were stopped for some reason) you need to use this script. 

If everything has gone correctly, you should see something akin to the following in your syslog:

Oct 20 16:00:12 dns1 mysqld: Version: '10.3.10-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 2 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 0 [Note] WSREP: Service thread queue flushed.
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 3a37c9d4-d481-11e8-99db-2e76f0c7ecda:0
Oct 20 16:00:12 dns1 systemd: Started MariaDB 10.3.10 database server.
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 2 [Note] WSREP: Synchronized with group, ready for connections
Oct 20 16:00:12 dns1 mysqld: 2018-10-20 16:00:12 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

You can also confirm my accessing the MySQL client and running:

MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 1 |
+--------------------+-------+
1 row in set (0.001 sec)

Now add the configuration to the other nodes, replacing the node names and IP addresses where appropriate.

Once that is done, restart MariaDB on nodes 2 and 3 normally. For example:

systemctl restart mariadb

If configured properly, they should each start up and join the cluster. You can confirm this by tailing the syslog on the first node. You’ll see lots of messages about the new node joining the cluster. 

Access the MySQL cli again, and confirm the cluster size:

MariaDB [(none)]> show status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.001 sec)

Congrats, you’ve got your Galera cluster up and running. To see it in action, try creating a database on node 1:

MariaDB [(none)]> create database test_database;
Query OK, 1 row affected (0.005 sec)

You’ll see it show up on node 2 right away:

[root@dns2 ~]# ls /var/lib/mysql/
aria_log.00000001 galera.cache ib_buffer_pool ib_logfile1 mysql tc.log
aria_log_control grastate.dat ibdata1 ibtmp1 mysql.sock test
dns2.pid gvwstate.dat ib_logfile0 multi-master.info performance_schema test_database

Unfortunately, although functional, this setup is terribly insecure. I recommend you take some steps to secure it, but that is outside the scope of this post. Check out http://galeracluster.com/documentation-webpages/security.html

Our next step is to setup PowerDNS to use this Galera cluster to serve DNS queries. 

Again, lets setup the PDNS repository so we can install the required software:

yum install epel-release yum-plugin-priorities &&curl -o /etc/yum.repos.d/powerdns-auth-41.repo https://repo.powerdns.com/repo-files/centos-auth-41.repo &&yum install pdns

yum install pdns-backend-mysql

We also need to setup the database that PDNS will use. Create the database:

MariaDB [(none)]> create database pdns;
Query OK, 1 row affected (0.005 sec)

Now execute the following SQL to setup the schema:

CREATE TABLE domains (
  id                    INT AUTO_INCREMENT,
  name                  VARCHAR(255) NOT NULL,
  master                VARCHAR(128) DEFAULT NULL,
  last_check            INT DEFAULT NULL,
  type                  VARCHAR(6) NOT NULL,
  notified_serial       INT DEFAULT NULL,
  account               VARCHAR(40) DEFAULT NULL,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE UNIQUE INDEX name_index ON domains(name);


CREATE TABLE records (
  id                    BIGINT AUTO_INCREMENT,
  domain_id             INT DEFAULT NULL,
  name                  VARCHAR(255) DEFAULT NULL,
  type                  VARCHAR(10) DEFAULT NULL,
  content               VARCHAR(64000) DEFAULT NULL,
  ttl                   INT DEFAULT NULL,
  prio                  INT DEFAULT NULL,
  change_date           INT DEFAULT NULL,
  disabled              TINYINT(1) DEFAULT 0,
  ordername             VARCHAR(255) BINARY DEFAULT NULL,
  auth                  TINYINT(1) DEFAULT 1,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX recordorder ON records (domain_id, ordername);


CREATE TABLE supermasters (
  ip                    VARCHAR(64) NOT NULL,
  nameserver            VARCHAR(255) NOT NULL,
  account               VARCHAR(40) NOT NULL,
  PRIMARY KEY (ip, nameserver)
) Engine=InnoDB;


CREATE TABLE comments (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  name                  VARCHAR(255) NOT NULL,
  type                  VARCHAR(10) NOT NULL,
  modified_at           INT NOT NULL,
  account               VARCHAR(40) NOT NULL,
  comment               VARCHAR(64000) NOT NULL,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE INDEX comments_domain_id_idx ON comments (domain_id);
CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);


CREATE TABLE domainmetadata (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  kind                  VARCHAR(32),
  content               TEXT,
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);


CREATE TABLE cryptokeys (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  flags                 INT NOT NULL,
  active                BOOL,
  content               TEXT,
  PRIMARY KEY(id)
) Engine=InnoDB;

CREATE INDEX domainidindex ON cryptokeys(domain_id);


CREATE TABLE tsigkeys (
  id                    INT AUTO_INCREMENT,
  name                  VARCHAR(255),
  algorithm             VARCHAR(50),
  secret                VARCHAR(255),
  PRIMARY KEY (id)
) Engine=InnoDB;

CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);

You can copy/paste it into an SQL file, or just curl it from https://blog.zswap.net/pdns.sql

[root@dns1 ~]# curl https://blog.zswap.net/pdns.sql | mysql -u root -p pdns
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2817 100 2817 0 0 12105 0 --:--:-- --:--:-- --:--:-- 12142
[root@dns1 ~]#

We also need to tell PDNS to use the MySQL back end. You’ll need to create a user for the PDNS database:

MariaDB [(none)]> grant all on pdns.* to 'pdns'@'localhost' identified by 'super_secure_password';
Query OK, 0 rows affected (0.172 sec)

Now open up your PowerDNS config file. The default location is /etc/pdns/pdns.conf. Scroll all the way to the bottom and add the following lines:

launch=gmysql
gmysql-host=localhost
gmysql-dbname=pdns
gmysql-user=pdns
gmysql-password=super_secure_password

Save the file and restart PDNS. 

[root@dns2 ~]# service pdns restart
Redirecting to /bin/systemctl restart pdns.service
[root@dns2 ~]#

Confirm that PDNS started up properly and connected to your Galera cluster:

Oct 20 16:31:18 dns2 systemd: Starting PowerDNS Authoritative Server…
Oct 20 16:31:18 dns2 pdns_server: Reading random entropy from '/dev/urandom'
Oct 20 16:31:18 dns2 pdns_server: Loading '/usr/lib64/pdns/libgmysqlbackend.so'
Oct 20 16:31:18 dns2 pdns_server: This is a standalone pdns
Oct 20 16:31:18 dns2 pdns_server: Listening on controlsocket in '/var/run/pdns.controlsocket'
Oct 20 16:31:18 dns2 pdns_server: UDP server bound to 0.0.0.0:53
Oct 20 16:31:18 dns2 pdns_server: UDPv6 server bound to [::]:53
Oct 20 16:31:18 dns2 pdns_server: TCP server bound to 0.0.0.0:53
Oct 20 16:31:18 dns2 pdns_server: TCPv6 server bound to [::]:53
Oct 20 16:31:18 dns2 pdns_server: PowerDNS Authoritative Server 4.1.4 (C) 2001-2018 PowerDNS.COM BV
Oct 20 16:31:18 dns2 pdns_server: Using 64-bits mode. Built using gcc 4.8.5 20150623 (Red Hat 4.8.5-16) on Aug 29 2018 14:13:06 by buildbot@944146f600c7.
Oct 20 16:31:18 dns2 pdns_server: PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
Oct 20 16:31:18 dns2 pdns_server: Polled security status of version 4.1.4 at startup, no known issues reported: OK
Oct 20 16:31:18 dns2 pdns_server: Creating backend connection for TCP
Oct 20 16:31:18 dns2 systemd: Started PowerDNS Authoritative Server.
Oct 20 16:31:18 dns2 pdns_server: About to create 3 backend threads for UDP
Oct 20 16:31:18 dns2 pdns_server: Done launching threads, ready to distribute questions

You’ll need to repeat this configuration on each Galera node you’d like to act as an authoritative nameserver. Two is preferred, three doesn’t hurt. 

Now let’s create a zone file to test this with:

[root@dns2 ~]# pdnsutil create-zone testdomain.com
Creating empty zone 'testdomain.com'

Set your editor, and edit it:

[root@dns2 ~]# export EDITOR=vim
[root@dns2 ~]# pdnsutil edit-zone testdomain.com

Unlike Bind, all domain names are exactly as they show. There is no need to add a trailing period to anything. Add a new record in the following format:

testdomain.com       300     IN      A       1.2.3.4

And quit out of the editor just like vim :wq

It will complain that there are errors with your zone, because we lack NS records. That is not an issue for our testing. Press “a” twice to accept and save the changes.

There is no need to restart PDNS and we should be able to test that its working right away:

[root@dns2 ~]# dig @localhost testdomain.com +short
1.2.3.4

And it will have already replicated to the other nodes in your Galera cluster. Try a DNS query against one of those:

[root@dns2 ~]# dig @pdns1.zswap.net testdomain.com +short
1.2.3.4

[root@dns2 ~]# dig @pdns3.zswap.net testdomain.com +short
1.2.3.4

Congratulations, you now have working authoritative nameservers and their zones replicated by Galera. At this point, you will need to figure out how you’d like to manage your zones.

Hint: pdnsutil is pretty useful.

Leave a Reply

Your email address will not be published. Required fields are marked *