OpenStack
Designate and External DNS
This is for OpenStack Train.
Overview
We have in our environment some existing split-horizon DNS services which serves data to "the public" (which may be actually public Internet facing or might be front-of-house in some otherwise closed environment). The point being is that those servers are front-facing and we, in OpenStack-controller-land, are not. Split-horizon is relevant as the same servers serve results to the public and to us, internally, but it requires a largely undocumented flag. sigh Not just that, because the flag is pool-specific then if we have a split-horizon server then we must have multiple pools.
Let's describe that in more detail (we could do with a scratchy diagram here but, hey ho). The public interacts with our existing front of house DNS servers and so everything those DNS servers do must be self-consistent and usable by the public. In particular, from the Designate-creating-zones perspective, the Source of Authority (SOA) and Name Server (NS) records must be referencing the front of house DNS servers. We cannot have Designate generating zone information which references hosts/IPs the public cannot reach. We'll tacitly assume that any subsequent resource records (RR) such as address (A and AAAA) records are ones that the public can also reach -- otherwise it's all a bit pointless.
The standard OpenStack Designate documentation does the easy thing and has Designate (running on the controller) use the controller (in particular, localhost) as the external DNS server. You don't really gain any insight, there. A little grovelling around can help us with getting started with Designate from which we get most of our clues.
Not that that means it is plain sailing but we'll cross the bridges in due course.
FWIW in the description below we are targeting Bind9 external DNS servers and so references to rndc and its port 953 are specific to that backend.
Designate
Designate is a collection of five services which collectively conspire to maintain a set of zones and RRs and poke the front of house DNS servers with updates and verify they have be transferred. There's a nice architectural diagram.
What happens in practice is quite subtle. You declare some pools in Designate where each pool is targeting a specific zone construction rules and external DNS tuple. Here arranging pools does get a bit messy but we'll come back to that.
For each pool, Designate will poke the described external DNS servers on port 953 saying "ask me (the controller) on port 5354 for an update to $ZONE". It will then query each (documented) external nameserver in turn to verify that it has been updated.
That's quite subtle and there's two things you should take from it:
- It is a two way interaction -- both parties will initiate a connection so adjust your firewalling as appropriate
- The external DNS servers and our controller must be able to route packets to one another.
That second point is relevant if you have carefully manufactured your Cloud's control plane to be hidden from prying eyes. To be fair, you already had to make your Cloud visible-ish, partly because you need to interact with either keystone, glance, nova etc. directly or through horizon web dashboard but also you probably use the VNC proxy to access the instance consoles through port 6080. That visibility is probably constrained to just your operational environment but that may now need extending to your external DNS servers.
Port 953 is the standard rndc control port and appears when you enable the controls attribute in named.conf. You can firewall it as appropriate and we will be using keys for trust. In order that Designate can talk rndc to your external DNS servers the bind-utils package will be installed on the controller. (The default installation also installs bind which is probably redundant in our case.)
Port 5354 is described as mDNS as in miniDNS as opposed to port 5353 which is mDNS as in multicast-DNS. Bind9 talks mDNS on port 5354 so we're good.
Installation
Installation is fairly straight-forward -- the fun starts when configuring pools -- although, because the example is only talking to an "external" DNS on localhost it completely misses any firewalling issues.
Firewall
Controller
The Cloud's control plane will to talking to the Designate service on port 9001.
The external DNS servers will be talking (inbound) to the controller on port 5354.
iptables -I INPUT -p tcp -m multiport --dports 9001 -m comment --comment "Designate incoming" -j ACCEPT iptables -I INPUT -p tcp -m multiport --dports 5354 -m comment --comment "Designate mDNS incoming" -j ACCEPT
Warning
This is not persistent.
Given that Train (still) abhors the use of firewalld on RHEL/CentOS then it is up to you to conceive a suitable systemd service to set this up on startup.
External DNS
The controller will be talking (inbound) to the external DNS on port 953 (rndc).
The controller will be verifying some/many/all of the external DNS name servers on port 53 -- you may well already allow this!
You might get away with:
firewall-cmd --add-port=953/tcp --permanent firewall-cmd --reload
Although you might want to restrict the source host/zone/... Remember we will be using keys so drive-by risks are mitigated a little.
Installation Tweaks
I'll mention it now but there's a couple of post-pool creation tweaks you might want to make to designate.conf.
In the Administration Guide for Designate there is a page on multiple pools which is of interest to us.
[service:central]
In the first instance change the default_pool_id to one of yours.
Secondly, the suggestion to set the scheduler_filters to *something*, fallback doesn't seem to work for me. Changing it to *something*, default_pool does do the right thing (even in the face of the default pool nonsense below).
The actual *something* is of interest and the examples given are for an explicit pool id being passed or by some attribute being passed. The really interesting option:
If we wanted to schedule the zone based on the tenant, we could write a custom filter that looked up the appropriate group and adds the appropriate pool.
has no further clues. sigh
To be fair, there is some documentation on writing pool schedulers although, having never added any code to OpenStack, I wouldn't know where to begin.
Train on RHEL/CentOS
I would note that for RHEL/CentOS the Train repo should be pure release 9.0.0 but it contains some historic release 8.0.0 rpms. That wouldn't normally be a problem except openstack-designate-pool-manager and openstack-designate-zone-manager are both no longer required in 9.0.0 but the suggested:
yum install -y openstack-designate-*
picks them up which in turn means there'll be clashes between the required versions of openstack-designate-common. How annoying. You can bodge your way through with a judicious:
yum install -y --skip-broken openstack-designate-*
and ignore the errors relating to openstack-designate-*-manager.
System Configuration
We need to do some more nuanced system configuration.
Keys
On the controller create an rndc key:
designate_key_name=cloudX-designate
rndc_designate_key_file=/etc/designate/rndc.key
rndc-confgen -a -k ${designate_key_name} -c ${rndc_designate_key_file} -r /dev/urandom
chown named:named ${rndc_designate_key_file}
I've left some variables in there from my scripting to help guide your own. designate_key_name is designate in the examples which will get confusing if you have more than one Designate service floating about. (Eh? You only have one cloud? Pfft!)
rndc.conf
Again, on the controller, create an rndc.conf (the documentation uses /etc/rndc.conf, the default config file for rndc) so that it can connect to the external DNS servers. Here we can encode the idea of talking to multiple external DNS systems by creating an rndc.conf per external DNS system:
# domain_master_ip=a.b.c.d
# cat <<EOF > /etc/rndc-${domain_master_ip}.conf
include "${rndc_designate_key_file}";
options {
default-key "${designate_key_name}";
default-server ${domain_master_ip};
default-port 953;
};
EOF
Again domain_master_ip supports the notion of multiple external DNS systems and should be the routable IP address of the external DNS server from the controller's perspective.
Note
Of interest, here, is default-server. In practice this file can be used for other external DNS servers in your pool file as Designate will explicitly set the server to be used.
When we sort out the external DNS servers we will be able to tickle them from the controller by running:
rndc -f /etc/rndc-${domain_master_ip}.conf cmd ...
If we had used the default config file, /etc/rndc.conf we could have simply said:
rndc cmd ...
Note
Both of the above will use the default-server address, of course. Add -s IP to target others.
named.conf
On all external DNS servers we need to edit named.conf in a few ways.
Note
I was a bit disappointed, here, as I thought we could just poke the master external DNS server and it would notify/update the slaves. Sadly, not. There'll be a squeak from the slave saying that the master is not authoritative (even though the zone file says it is).
Firstly, though, we need to copy over the key we created above. My script, running on the controller, tells me to run the following on each external DNS server (noting that the contents of the rndc_designate_key_file will be printed to screen for me to cut'n'paste):
(
umask 077
cat <<EOC > /etc/rndc-${designate_key_name}.key
$(cat ${rndc_designate_key_file})
EOC
)
So, something like:
(
umask 077
cat <<EOC > /etc/rndc-${designate_key_name}.key
key "${designate_key_name}" {
algorithm hmac-md5;
secret "nothingtoseehere==";
EOC
)
Remember here that I used a value for designate_key_name which identified the Cloud/controller I was using. If you are just using the value designate as per the documentation then you can handle multi-Designate conflicts yourself.
For named.conf itself we need to make several edits.
When Designate pokes this external DNS server with rndc it will be using the addzone and delzone commands. These require the allow-new-zones option to be enabled.
The documentation says we require IXFR (as opposed to AXFR) to be disabled. Not sure what the reason is (maybe mDNS doesn't support it?) but let's do it:
options {
...
allow-new-zones yes;
request-ixfr no;
...
};
Next we need to allow the controller to talk to us at all. To do this we need to turn on the rndc listener (controls { inet $IP ...}; resulting in Bind listening on port 953 on $IP). We then constrain that to allowing access to ourselves, ${IP}, (probably redundant but I suppose we can poke about) and the controller's accessible IP (the routable IP we mentioned earlier) when using our key, ${designate_key_name}:
include "/etc/rndc-${designate_key_name}.key";
controls {
inet ${NS_IP}
allow { ${NS_IP}; ${controller_access_ip}; }
keys { "${designate_key_name}"; };
};
Finally, with our split-horizon hats on we need to indicate which view is allowed to be manipulated. In this case a view called lab:
view "lab" {
...
allow-update { ${NS_IP}; ${controller_access_ip}; };
};
although you might want to be more precise with:
view "lab" {
...
allow-update {
{ ${NS_IP}; key "my-local-key"; };
{ ${controller_access_ip}; key "${designate_key_name}" };
};
};
and you may want to look at allow-update-forwarding and allow-transfer and whatever else is required.
The allow-update statement uses the same ACL as above.
Finally, a few bits of housekeeping:
chown named:named /etc/rndc-${designate_key_name}.key
systemctl restart named
firewall-cmd --add-port=953/tcp --permanent
firewall-cmd --reload
Pools
The Default Pool
I would suggest that you don't use this as it only confuses the issue. However there is something far more subtle about the default pool.
It will be created when you first run (subsequent runs will not recreate it):
su -s /bin/sh -c "designate-manage pool update" designate
and takes the id 794ccc2c-d751-44fe-b57f-8894c9f5c842. How can I know that? Because it is hard-wired!
Not only is it hard-wired it is also required. Not in a "fatal if it is missing" required way but a "you're joking" required way. Here's how:
# designate-manage pool show_config
Pool Configuration:
-------------------
also_notifies: []
attributes: {}
id: 794ccc2c-d751-44fe-b57f-8894c9f5c842
name: default
nameservers: []
ns_records: []
targets: []
show_config looks for the pool with id 794ccc2c-d751-44fe-b57f-8894c9f5c842. It does not look for the pool called default, it does not look for the default_pool_id as defined in designate.conf. Other things do use default_pool_id, but not show_config. sigh
show_config --pool_id $ID does do the right thing although at this stage you don't know what $ID is.
Of note is that if you try to create a zone, now, before you've done anything else:
# openstack zone create --email hostmaster@runscripts.com example.runscripts.com. no_servers_configured
A little bit has happened under the hood, here, but it eventually boils down to the value for ns_records in the chosen pool is empty. Which it is, for the default pool, as seen above.
If you do delete the default pool (not advised) by firstly removing the entry for default from /etc/designate/pools.yaml then:
designate-manage pool update --delete foo
(an argument is required but not seemingly used...)
then subsequent show_config commands (without a specific --pool_id) will spew unhelpful errors down your page.
Not only that but you now cannot recover that default pool. If you re-add the details above back into pools.yaml (without the id flag as you'll discover it isn't allowed for new pools) and re-run:
designate-manage pool update
then show_config still barfs even though there is a default pool again (but now with a new id).
Database Repair
While we're passing through we can touch on database repair. As with all "poking about in the DB" we're second guessing what the developers are doing so, er, good luck to us.
Broken Zone Creation
You can stop the repeated attempts by Designate to signal the external DNS that something needs adding or deleting by squishing the zones in the DB. Something like the following seems to work:
# mysql -u root -p designate mysql> update zones set deleted=id,deleted_at=NOW(),status='DELETED',action='NONE'; mysql> \q # systemctl restart designate-*
(although you might want to be more prudent with some suitable WHERE clause. Or not.)
Restarting
If all hell breaks loose you can always drop the database and restart. The nuclear option:
# mysql -u root -p mysql> drop database designate; mysql> create database designate; mysql> GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'localhost' IDENTIFIED BY 'DESIGNATE_DBPASS'; mysql> GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'%' IDENTIFIED BY 'DESIGNATE_DBPASS'; mysql> \q # designate-manage database sync # systemctl restart designate-*
which, as a side effect, will re-create the default pool with id 794ccc2c-d751-44fe-b57f-8894c9f5c842.
New Pools
OK, let's review what we need to make the following work:
A pool name (which we can't change -- although we can delete it which, for everything except default should work although remember that it will delete any zones in it)
A set of name servers for the created zones, ns_records. These are going to be the NS records in the zone.
Like all NS records these must be a .-terminated domain name (and not an IP address) and must exist outside of Designate's control. This is where you may require to have populated the parent domain with some glue records.
Of note, here, is that each entry has a priority attribute which should work to the benefit of the higher numbered version. In principle you can describe the ns_record entries in any order so long as the future SOA record has the highest priority (largest number).
A set of name server IP addresses to verify the creation/update has occurred, nameservers. These are probably the IP addresses of the same set of name servers as the ns_records above.
Finally a set of target external DNS servers to poke.
Again, noting my wishful thinking about being able to get away with just poking the master external DNS server and have it update the slaves, this set is almost certainly the same set of servers as in the previous two cases.
The configuration here includes external DNS configuration information (for Bind9 that will be some rndc options including a key) plus some information about us, the controller, who is the nominal master for the zone.
If you have a split-horizon setup in your external DNS service then you will need an addition (largely undocumented) rndc option called view
What does that lot look like? With a bit of parameterising we get something like:
- name: ${pool_name}
description: ${pool_name} Pool
attributes: {}
ns_records:
- hostname: ${FQDN_NS1_record}
priority: 10
- hostname: ${FQDN_NS2_record}
priority: 1
nameservers:
- host: ${NS1_IP}
port: 53
- host: ${NS2_IP}
port: 53
targets:
- type: bind9
description: BIND9 Server NS1
masters:
- host: ${controller_access_ip}
port: 5354
options:
host: ${NS1_IP}
port: 53
rndc_host: ${NS1_IP}
rndc_port: 953
rndc_key_file: ${rndc_designate_key_file}
rndc_config_file: /etc/rndc-${domain_master_ip}.conf
view: '${bind9_view}'
- type: bind9
description: BIND9 Server NS2
masters:
- host: ${controller_access_ip}
port: 5354
options:
host: ${NS2_IP}
port: 53
rndc_host: ${NS2_IP}
rndc_port: 953
rndc_key_file: ${rndc_designate_key_file}
rndc_config_file: /etc/rndc-${domain_master_ip}.conf
view: '${bind9_view}'
Where bind9_view is lab, of course.
Warning
FQDN_NS1_record must be the properly qualified name server record. Unless you really know what you are doing then that means it must end with a . -- ns1.example.com. for example.
As per the proper rules of a zone, if any name does not end in a . the the name of the zone itself is appended. Of course, I'm teaching you to suck eggs, here.
Update / Check
The first time we ran designate-manage pool update it created the default pool (amongst other things). From now on we can use it to create / update and delete pools.
The normal action is that any pool described in /etc/designate/pools.yaml (or by the --file YAML argument) are created (if not already present) or updated otherwise. Only if you pass the --delete flag will it delete pools not in the YAML file.
Note
In Train --delete requires an argument which doesn't seem to get used for anything. You would like to think it identified the pool you wanted to delete. Maybe in the next release.
Unfortunately you don't get to see what update has done until you ask it to generate a file:
designate-manage pool generate_file
which stomps over /etc/designate/pools.yaml unless you pass --file YAML.
If you have added a new pool you may want to restart the Designate services just to be sure:
systemctl restart designate-*
And, FWIW, the most useful file, from our external DNS perspective, to keep an eye on is /var/log/designate/worker.log.
Zone Creation
Enough idle banter, let's do something.
Warning
cave exemplum
"Muahaha!" You are thinking, "Let's create example.com.!" Woo!
Well, you're about to get bitten.
The problem isn't in the creation of example.com., which will work fine, but in the deletion of it. Why? Well, the problem lies in how Designate determines that the external DNS has done something. It queries the external DNS servers for the SOA of a domain and checks the serial number. That's pretty reasonable for the creation side of things and it will successfully check that the serial number in the SOA is the number (Unix time_t it appears) it first thought of.
If it doesn't get the correct serial number then /var/log/designate/worker.log will report:
Could not find $SERIAL for $ZONE on enough nameservers.
which seems reasonable though I have seen it use 0 for $SERIAL leading to the slightly cryptic:
Could not find 0 for $ZONE on enough nameservers.
Which I eventually tracked down in the source to figure out what it was complaining about.
Anyway, when we come to delete the zone we would expect that any request for an SOA to fail -- because the zone has gone away, right?
Bah! In the case of example.{com,org,net}. we're bitten by the fact that the IANA has real RRs for those zones and if your external/front of house DNS servers are wired up such that they can query the Internet-proper then they will resolve the real example.com.'s SOA record and get back some serial number. It won't be the one Designate thought of but more importantly it won't be NXDOMAIN.
So Designate will put the zone into a pending deletion/error state, erm, forever.
Of course I'm teaching you to suck eggs (again) here but you should only be using domains (and sub-domains thereof) you own. There are no local/test/example domains.
As to how you mitigate tenants either sniping each others domains or shadowing others there's some suggestions in the Production Guidelines but it's a best efforts game.
On your external DNS servers you may want to:
tail -f /var/named/data/named.run
(on RHEL/CentOS at least)
and watch for interesting traffic.
On some box that you would normally run openstack commands from try (using a domain you own!):
openstack zone create --email hostmaster@runscripts.com example.runscripts.com.
named should have spat out something along the lines of:
ns1 named[23569]: received control channel command 'addzone example.runscripts.com in lab { type slave; masters { ${controller_access_ip} port 5354;}; file "slave.example.runscripts.com.6087122b-5e64-42fd-acdd-26ad83d98cf7"; };'
ns1 named[23569]: added zone example.runscripts.com in view lab via addzone
ns1 named[23569]: zone example.runscripts.com/IN/lab: Transfer started.
ns1 named[23569]: transfer of 'example.runscripts.com/IN/lab' from ${controller_access_ip}#5354: connected using ${NS1_IP}#37226
ns1 named[23569]: client @0x7f73345d30f0 ${controller_access_ip}#42461: view lab: received notify for zone 'example.runscripts.com'
ns1 named[23569]: zone example.runscripts.com/IN/lab: notify from ${controller_access_ip}#42461: refresh in progress, refresh check queued
ns1 named[23569]: zone example.runscripts.com/IN/lab: transferred serial 1584122472
ns1 named[23569]: transfer of 'example.runscripts.com/IN/lab' from ${controller_access_ip}#5354: Transfer status: success
ns1 named[23569]: transfer of 'example.runscripts.com/IN/lab' from ${controller_access_ip}#5354: Transfer completed: 1 messages, 4 records, 172 bytes, 0.010 secs (17200 bytes/sec)
ns1 named[23569]: zone example.runscripts.com/IN/lab: sending notifies (serial 1584122472)
Cool. Notice the GUID-appended zone name, slave.example.runscripts.com.6087122b-5e64-42fd-acdd-26ad83d98cf7. That's now a real file in /var/named (albeit the contents are rather cryptic).
We can check that from the controller with rndc:
# rndc -c /etc/rndc-${domain_master_ip}.conf showzone example.runscripts.com. IN ${bind9_view}
zone "example.runscripts.com" in lab { type slave; file "slave.example.runscripts.com.6087122b-5e64-42fd-acdd-26ad83d98cf7"; masters { ${controller_access_ip} port 5354; }; };
Looks good. Let's check the zone's details -- check against all of your external name servers (NS1_IP, NS2_IP, ...):
# host -a example.runscripts.com. ${NS1_IP}
Trying "example.runscripts.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16107
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 2
;; QUESTION SECTION:
;example.runscripts.com. IN ANY
;; ANSWER SECTION:
example.runscripts.com. 3600 IN SOA ${FQDN_NS1_record} hostmaster.runscripts.com. 1584122472 3584 600 86400 3600
example.runscripts.com. 3600 IN NS ${FQDN_NS2_record}
example.runscripts.com. 3600 IN NS ${FQDN_NS1_record}
;; ADDITIONAL SECTION:
${FQDN_NS2_record} 3600 IN A ${NS2_IP}
${FQDN_NS1_record} 3600 IN A ${NS1_IP}
Received 168 bytes from ${NS1_IP}#53 in 1 ms
Excellent. In particular the SOA record is using FQDN_NS1_record. Obviously the NS records come out in some order and will probably be a different order the next time you ask.
Zone Deletion
No biggie:
openstack zone delete example.runscripts.com.
with named reporting:
ns1 named[23569]: received control channel command 'delzone example.runscripts.com in lab' ns1 named[23569]: deleting zone example.runscripts.com in view lab via delzone ns1 named[23569]: zone example.runscripts.com scheduled for removal via delzone
and from the controller:
# rndc -c /etc/rndc-${domain_master_ip}.conf zonestatus example.runscripts.com. IN ${bind9_view}
rndc: 'zonestatus' failed: not found
no matching zone 'example.runscripts.com.' in view 'lab'
Top!
Multiple Pools
Creating multiple pools is trivial. Indeed, as we noted above, because we have a split-horizon external DNS server and any view requires a distinct pool then we are required to have multiple pools.
The real question is, how do I use them, or, more correctly, how do I choose between them? An even better question might be which pool is OpenStack (neutron in particular) going to use?
As mentioned before, in the Administration Guide for Designate there is a page on multiple pools which notes a couple of possible schedulers, pool_id_attribute and attribute.
If you know your pool id (hint: run generate_file as above and read the file) you can easily select a pool on the command line:
openstack zone create ... --attributes pool_id=$GUID
If you want to use the attribute scheduler then you'll have to define some attributes in your pools. Follow the example in the link. Something like (noting that the attributes are all arbitrary strings, here the-pool and the-view are meaningful to us):
attributes:
the-pool: ${pool_name}
the-view: ${bind9_view}
and use it as above:
openstack zone create ... --attributes the-pool:${pool_name} the-view:${bind9_view}
as you will quickly discover it is
Attribute 'the-pool=${pool_name}' is in an incorrect format. Attributes are <key>:<value> formated[sic]
Note
Using both a pool-distinguishing attribute and a view-distinguishing attribute means you can select the pool more precisely than just a single attribute alone.
Of course you could use a compound attribute:
attributes:
unique-name: ${pool_name}-${bind9_view}
The real question is how do we get the horizon web interface to use a particular pool? It will be using the default_pool scheduler, ie. whatever you have set default_pool_id to be in designate.conf. Not very flexible.
We referenced the OpensStack page on writing pool schedulers above.
designate.conf
designate.conf does reference a couple of OpenStack integrations: [handler:neutron_floatingip] and [handler:nova_fixed] both mention an explicit zone_id attribute which seems a bit specific.
You can set a dns_domain on a neutron network and you can set both a dns_domain as well as a dns_name on a neutron port so you would think that that would give Designate some clues but it won't help you if the same DNS domain name appears in multiple views (internal and external, say).
Conclusion
Designate can handle multiple pools, OpenStack cannot. Even if we wrote a pool scheduler, OpenStack is only going to drive one pool and therefore only one Bind9 view.
So, if we want to integrate OpenStack/Designate into our external-facing DNS then that DNS services must be a single view service (or at least a single view that captures traffic from both internal and external traffic). It cannot be a split-horizon service.
Document Actions