2019.2.2 Hosts with IPv4 and IPv6 can't talk to IPv4 master #55214
Comments
|
(7361:6c74:2e6d:7870:7269:6d65:2e6e:6574) |
|
Is that a typo in your response? That's not what was specified on the command line, nor does that string exist anywhere on the minion.
The minion was communicating with the master right up until I ran I can telnet from the salt minion to the master on ports 4505 and 4506 and receive data. |
|
looks like salt is resolving the dns name to that ipv6 address: do you have |
|
Yes, that's what it looks like, but... No IPv6 address for the salt server in DNS. I can't figure out where it's coming from. And I can connect to both ports with telnet. |
|
can you share a sanitized version of your config? |
|
The The I ended up wiping the box, disabling IPv6 and re-provisioning it since it was under salt anyways... ;) So no longer have an environment to test or confirm that the issue was resolved. I'd say it can be closed unless someone else can reproduce it. |
|
I was able to duplicate it on a different machine talking to the same master and gather additional information. My salt master does have an IPv6 address. It is NOT published in DNS. This: IPv6 address doesn't appear to exist on either the minion or the master. Minion: Master: I have 70 hosts connected to this master without any problems. I can't find anything unusual about this particular minion that separates it from the rest. It's part of 16 identical boxes (hardware and software) that have a Debian 9 install and are entirely configured by Salt, with the only differences being their IPv4 addresses. They are all behind firewalls that are also configured with Salt and the only difference with the firewalls is their IPv4 private IPs and their IPv4 public IPs. They are all connected via Comcast. Anything else I can do to debug? |
|
apologies i was not clear, can i see your master and minion sanitized config? |
|
Salt master config: Salt minion config: |
|
I just double-checked both the master and minion and made sure they had |
|
I've tried deleting everything under I've also tried clearing the DNS cache and double-checked from several boxes around the internet that there is indeed no IPv6 record for my salt master. I even went as far as grepping for the IPv6 address under /etc, /var, and /usr. Nothing found. I've looked on the upstream DNS server and no DNS query ever comes in. In this screenshot .240 is the minion, and .254 is the DNS server:
Showing there are now AAAA records: running I'm completely stumped as to where that address is coming from. |
|
I had a bunch of changes I needed to get to this box since it's in production. The OS is entirely configured by salt and the data resides on a different drive--so I remove salt, purged the packages, nuked /etc/salt, did a bunch of funky |
|
Found a third box with this issue running salt 2019.2.1. |
|
Well...ran into another box with the issue. There is no ipv6 connectivity at this location. |
|
I've got another box with this issue. It was working fine years. It doesn't appear to be dependent on location (multiple boxes across different states), client (it's occurring on 3 different client networks that are all wildly different), internet connection (one site is Comcast, another is Charter, a third is Wave Broadband). DNS has no IPv6 record for salt.mxpriem.net. The box receives most of its traffic from a port forward on the firewall, so I disabled it and ran wireshark so I could more easily filter through the traffic. Similar to the screenshot I posted above, it does a AAAA query for salt.mxprime.net, gets NO answers back, doesn't even bother doing an A query, and then magically generates tan IPv6 address out of thin air. The strange thing about the IPv6 address is that it doesn't seem to matter which box, what OS version, what random DNS server it's pointing to or any of the other info...it always returns that same IPv6 address. If I run I'm debating what I should try next because I'm pretty much stumped at the moment. I haven't tried switching versions. The box was running 2019.2.3 and I upgraded it to 2019.2.5 and it didn't fix the problem. I'm debating bumping up to 2020.x. I haven't tried spinning up a completely unrelated DNS zone hosted on a separate provider with different records pointing at 107.170.241.41 to see if it's possibly some bizarre DNS issue....although the packet capture showed nothing in the response. Every other program can communicate using salt.mxprime.net with no issue. I spun up a temp webserver and curled down a file. I ran netcat and connected to it with no problems. I'm completely freaking stumped as to where this IPv6 address is coming from...because only salt seems to be affected on this box... ...but here's the real kicker. For fun, I ran The current box that is having issues is scheduled to be decommissioned within a month, so I'm open to aggressive troubleshooting. I mean I already nuked every file on the box with 'salt' anywhere in the name, so... ;) |
|
Does the following work for you? I have a strong feeling your stub resolver on minion with IPv6 tries to resolve master hostname, gets the FQDN from it and tries to connect to IPv6 first, as IPv6 is disabled on master it fails. In order to check it you might run from minion btw, did you check for AAAA records? Another thing I'd try is to fire up tcpdump to check exactly what's going on with minion |
Yeah--if I use the IPv4 address directly, it has no problems communicating with the master.
The minion has no IPv6 connectivity. Every other tool on the box definitely connects via IPv4.
I already did. If you see previous comments I posted a screenshot of the DNS traffic from the minion. It definitely requests an AAAA record, but it doesn't get an answer back. It never queries for an A record that I can see. |
|
ok, what if you use |
|
I tried setting that on the master as well as the |
|
That's absolutely weird, as well as this custom IPv6 minion tries to connect to. Do you use active directory or multiple IPs for the same hostname? Are you able to check DNS zone? |
|
I manage the zone mxprime.net. I have a DNS server running BIND. It's an extremely simple zonefile... No IPv6. The minion is on a private network that has AD, but it's not pointed at an AD DNS server. I've even tried setting resolv.conf to look directly at my BIND server...it still gets that IPv6 address. You're right...this is baffling. |
|
I have the exact same problem after I removed ipv6 from a working setup (moved the VM's to another provider that does not provide me with an ipv6 range):
|
|
Can the issue be reproduced on the latest version of salt? |
|
I'm running salt-3000.3-1.el7, which is the latest version in my CentOS7 repo's. |
|
I'm running salt 3000.x across most of my infrastructure now and haven't run into this in a while. I never found the root cause, but it hasn't happened to me in over a year. Unless someone else is having this problem, I'm going to close this ticket as I can no longer reproduce it to help troubleshoot. Let me know if I need to re-open it. |
Description of Issue
The majority of my hosts are IPv4 only including my master.
I upgraded one host that has both IPv4 and IPv6 running to 2019.2.2.
It can't communicate with the master and it appears to be getting an IPv6 address out of thin air.
Steps to Reproduce Issue
The thing I can't figure out is that the IPv6 address listed (7361:6c74:2e6d:7870:7269:6d65:2e6e:6574) is nowhere to be found.
It's not on the minion. There are no IPv6 DNS records for the master.
Versions Report
Maybe something to do with changes in #54762?
The text was updated successfully, but these errors were encountered: