Need help with configuring PowerDNS as secondary for zone when master server != notifying server

[irishman59]

PowerDNS Authoritative Server, v4.6.4

Running in primary+secondary mode.

primary=yes
secondary=yes

UPDATED: allow-notify-from is default (0.0.0.0/0)

Most of the zones on server are primary, but sometimes we need to add secondary zones from external DNS servers (which we don't control, because it's our customers)

And now we got this case:

• Secondary zone example.com
• Master servers for example.com (which are allowed to serve AXFR requests) are 1.1.1.1 and 2.2.2.2
• And separate server, which is not serving AXFR, but only sends NOTIFY about example.com zone changes - 3.3.3.3

And now we trying to configure our PowerDNS server to work, and we are getting problem, when customer changing (add/edit/delete) records in his zone example.com, and sending NOTIFY about it to our servers.

Scheme 1
If we add all servers (1.1.1.1, 2.2.2.2, 3.3.3.3) as Primaries for example.com, then:

1. NOTIFY are working fine
   Received NOTIFY for example.com from 3.3.3.3 - queueing check
2. but when our server goes after zone example.com using AXFR, it sometime (33%, 1 of 3 servers) trying to get zone from 3.3.3.3, which is not serving AXFR requests, and we are getting errors like:
   Unable to retrieve SOA for example.com, this was the first time. NOTE: For every subsequent failed SOA check the domain will be suspended from freshness checks for 'num-errors x 60 seconds', with a maximum of 3600 seconds. Skipping SOA checks until 1699390010
   Eventually at one of next retries our server gets to 1.1.1.1 or 2.2.2.2 (not 3.3.3.3), and receive updated zone.

XFR-in zone: 'example.com', primary: '1.1.1.1', initiating transfer
XFR-in zone: 'example.com', primary: '1.1.1.1', starting AXFR
AXFR-in zone 'example.com', primary '1.1.1.1', retrieval started
AXFR-in zone: 'example.com', primary: '1.1.1.1', retrieval finished

Scheme 2

If we add only servers 1.1.1.1, 2.2.2.2 as Primaries for example.com, then:

1. NOTIFY not working at all, because 3.3.3.3 is not listed as master server for the zone.
   Received NOTIFY for example.com from 3.3.3.3 which is not a master (Refused)

2. But AXFR requests when serial is stale always working (as we never try to AXFR from 3.3.3.3)

Domain 'example.com' is stale, master 1.1.1.1 serial 2023110802, our serial 2023110702
XFR-in zone: 'example.com', primary: '1.1.1.1', initiating transfer
XFR-in zone: 'example.com', primary: '1.1.1.1', starting AXFR
AXFR-in zone 'example.com', primary '1.1.1.1', retrieval started
AXFR-in zone: 'example.com', primary: '1.1.1.1', retrieval finished

Both schemes are partly working, but are there any method to get it working as expected -- allow getting NOTIFY for example.com not only from masters, but also from separate server?

[irishman59]

Only thing I found regarding this matter - it's this issue (feature request) #8816 , and it's still 'open' :(

[franklouwers]

As an alternative, if they can't send a NOTIFY from a host which can accept an AXFR, their updates will be slower, but will still based based on the Refresh timers.

[irishman59]

Sure, it'll work.

We are sticking to "Scheme 1" now, because eventually one of retries hits servers, which are accept AXFR, and getting zone update after all (despite of periodic errors and cooldown timer). It's even faster when rely only on Refresh timers on "Scheme 2"

But I try to understand now, are there any "correct way" to do all of this?

Or it's unsupported scheme for PowerDNS, when NOTIFY comes from one server, and AXFR served by another?

[franklouwers]

the "correct way" would be for the ip sending the notifies, to allow the axfr :)

[irishman59]

Oh, if only :) Then this whole discussion wouldn’t exist.

[irishman59]

BTW, looks like Bind has option like that, but unfortunately I cannot check right now if it's true and working as described (since our DNS servers are running on PowerDNS only :)

https://www.zytrax.com/books/dns/ch7/xfer.html#allow-notify

allow-notify

allow-notify { address_match_list };

allow-notify applies to slave zones only and defines a match list, for example, IP address(es) that are allowed to NOTIFY this server and implicitly update the zone in addition to those hosts defined in the masters option for the zone. The default behaviour is to allow zone updates only from the masters IP(s). This statement may be used in a zone, view or global options clause.

[klaus-nicat]

You can use the global setting allow-notify-from. That way, that customer's IP could also send NOTIFYs for other customer's domains. So customer A can trigger "refresh-checks" for zones of customer 2. That is not nice, but does not harm.

Of course, #8816 would be much nicer.

Another "hack" would be NAT. Maybe you could implement in iptables something like "if src=3.3.3.3 SRCNAT to 2.2.2.2".

[franklouwers]

@klaus-nicat as the default for allow-notify-from (per docs) is 0.0.0.0/0 (+ipv6 equiv), how would this change anything?

[irishman59]

Yep, @franklouwers is right, allow-notify-from in our configs is default -- 0.0.0.0/0 (+ipv6 equiv)

Another "hack" would be NAT

@klaus-nicat , thanks, nice idea!
But it's pretty dirty hack, I think. It will be hard to manage, if more schemes like this come by.
Maybe as last resort, if no "right powerdns way" provided.

[klaus-nicat]

Sorry, I confused trusted-notification-proxy with allow-notify-from.

[irishman59]

@pieterlexis idea (from IRC channel) about setting "3.3.3.3" (NOTIFY server) as trusted-notification-proxy on our servers saved the day!

And now "scheme 2" described in first post working fine.

# pdnsutil show-zone example.com
This is a Slave zone
Primaries: 1.1.1.1:53 2.2.2.2:53
Last time we got update from primary: Wed 2023-11-08 20:05:39
SOA serial in database: 2023110814
Refresh interval: 10800 seconds

# cat /etc/pdns/pdns.conf | grep proxy
trusted-notification-proxy=3.3.3.3/32

Log output:

Received NOTIFY for example.com from trusted-notification-proxy 3.3.3.3
Received NOTIFY for example.com from 3.3.3.3
Received NOTIFY for example.com from 3.3.3.3 - queueing check
Got NOTIFY for example.com, going to check SOA serial
Domain 'example.com' is stale, master 1.1.1.1 serial 2023110806, our serial 2023110804
XFR-in zone: 'example.com', primary: '1.1.1.1', initiating transfer
XFR-in zone: 'example.com', primary: '1.1.1.1', starting AXFR
AXFR-in zone 'example.com', primary '1.1.1.1', retrieval started
AXFR-in zone: 'example.com', primary: '1.1.1.1', retrieval finished
AXFR-in zone: 'example.com', primary: '1.1.1.1', storage transaction started
AXFR-in zone: 'example.com', primary: '1.1.1.1', zone committed with serial 2023110806

IRC chat part for historical purposes:

> <lieter> irishman59: you might need to set the trusted-notification-proxy setting: https://doc.powerdns.com/authoritative/settings.html#setting-trusted-notification-proxy
> <lieter> irishman59: I believe that pdns by default will only try the server that sent it the NOTIFY
> <lieter> irishman59: and don't set 3.3.3.3 as the primary
> <ottom> though I wonder how it handles trusted-notification-proxy ... whcih IP will it contact for AXFR?
> <lieter> irishman59: and you should have 3.3.3.3 in allow-notify-from
> <lieter> ottom: I believe the zone is marked for checking
> <lieter> ottom: so the next time the cycle-interval triggers it will contact a primary
> <ottom> the docs for trusted-notification-proxy are very lightweigth
> <lieter> ottom: it is. I believe the manpage for nproxy has more info on what a proxy is: https://doc.powerdns.com/authoritative/manpages/nproxy.1.html
> <ottom> irishman59: can you try what lieter suggested?
> <irishman59> lieter: "you might need to set the trusted-notification-proxy" , I'm afraid because of lack of documentation on this parameter, I don't fully understand it's usage and possible impact. Should I add '3.3.3.3' to it? Doesn't it allow to this server NOTIFY for all zones, not only example.com?
> <lieter> 3) yes it will allow 3.3.3.3 to notify for all your domains, but there are checks in place (pdns will check the SOA of the zone at its configured primaries)