Recently I have been working on a couple of new ISE deployments for our customers and noticed the same ISE alarm (Active directory diagnostic tool found issues) was firing in both environments.
Clicking on the above link, show us that the test “DNS SRV record query” is the one that is causing the issue.
Selecting the link under the “Results and Remedy” column gives us a little more information but still was not clear to me what is going on here. Where is the remedy the above link promised?
Ok so reading the alert (SRV record found. Not all SRV records have IP, will need to run additional query for get IP.) and ignoring that “for” should be “to” it seems to me that this is more of an optimization than a real issue. The alert is saying that the SRV record has been found but not all the data has been returned meaning ISE will need to run additional DNS queries to get the remaining data. I am not sure this alarm deserves a “Warning” classification level when an “Informational” alert would suffice in my opinion, but a warning it is, and I don’t like seeing warnings on my ISE home page.
Before sending myself off down a rabbit hole I asked a few peers if they had seen this same alert on any of their deployments and the answer was a resounding yes, although no one had an explanation for what caused it. Right, well its 2020 and I am struggling to make sense of a lot of things going on in our world right now, but I was not going to add this one to the list!
Time to dig in. First step is to take a peek at our ISE AD diagnostic tool page (Administration->Identity Management->External Identity Sources->Active Directory->Advanced Tools->Diagnostic Tool)
From the below screenshot we can see that all our tests are successful with the exception of our “DNS SRV record query”. We can also see that that we are running our tests daily at midnight, which is why our alarm counter increases each day. I started a packet capture on our ISE node and ran the single failing test manually by selecting it and hitting the “Run Test” button.
The packet capture below shows us that our ISE server 10.1.20.60 is querying its configured DNS server 10.1.20.9 for the SRV resource record “_ldap._tcp.dc._msdcs.lookingpoint.com”. This resource record is used to advertise all the domain controllers in the lookingpoint.com domain that are running the ldap (tcp port 389) service. I have also highlighted (red box) that the additional resource records have not been set in this query, indicating that this is not and EDNS ("extension mechanisms" for DNS) query. More on EDNS later.
Looking at the response I noted that there were nine answers (green box) and only three additional resource records (red box). Normally we would expect to see at least the same if not more additional records than answers
So, we can now see that we are missing additional records from the above response which explains our alert, but the question remains why our DNS server is not responding with all the data. The answer to this lies in the DNS RFC 1035. As the excerpt below from the RFC states, DNS UDP messages should not exceed 512 Bytes. Looking again at our response packet above we can see that the frame length (purple box) is 542 Bytes. 542 minus 14 Bytes (ethernet header), 20 Bytes (IP header) and 8 Bytes (UDP), leaves us with a DNS header and payload of 500 Bytes.
Taking a closer look (see below) at the additional records section of our DNS response we can see that each record consumes 16 Bytes. So, a fourth record would take our DNS size to 516 (500 + 16) Bytes which is greater than the maximum allowed of 512.
Interestingly since this DNS response is truncated the DNS server should be enabling the TC (truncated content) flag, but this is not the case with our response (blue box). Had this flag been checked then ISE would have rerun the query using TCP which can support larger sizes. Not sure why our Windows 2016 DNS server was not doing this, but I’ll leave that question for you windows guys and gals out there to answer.
Next, I wanted to see if our Windows DNS server supported EDNS. EDNS allows for extensions to be made to the original DNS standard. One such extension is to allow UDP responses bigger than the 512 Byte limit we encountered above. Testing using the Windows built in “nslookup” utility I received the same response that ISE (9 Answers and 3 Additional).
But using a different DNS command tool “Dig” I was able to craft a query with EDNS support, and the answer was complete. Note the 11 additional records and the size of 623 Bytes.
Looking at a packet capture when using the dig command above we can see that query has one additional resource record (green box), indicating the clients support of UDP payloads up to 4096 Bytes. This was absent from our ISE query.
Finally, below shows our server response and its support for EDNS (purple arrow) and DNS header size of 623 (purple box)
Conclusion
If your domain has a ldap srv records that take up more than 512 Bytes then you will see this alert.
ISE up to version 2.6 does not support EDNS in its queries which would resolve this issue. I’ll have to test again when im on a 2.7 or 3.0 build to see if this issue still exists. Sadly, there is not much we can do to suppress this alarm other than turning it off all together, which I do not recommend as it disables all the AD connector alerts.
Additional Note
For those of you with a keen eye you may have noticed that the nine answers received in our test environment above, contained some duplicate entries (UPPER and lower case). For example, we see LPDC01 and lpdc01. This domain only has five domain controllers, so we should only be seeing five answers. A quick google search brought up the below Microsoft article which I believe may account for this. I’ll have our internal Windows team take a look and update this post once the fix has been applied.
Windows DNS registers duplicate SRV records for a DC if its computer name has uppercase letters
As always if you have any questions on getting Cisco's ISE set up for you and your business and would like to schedule a free consultation with us, please reach out to us at sales@lookingpoint.com and we’ll be happy to help!
Check out our awesome tech talk on ISE: