There are a few things to check if your users are having difficulty connecting through Gateways to the (protected) hosts.
It is important to understand the differences between IdP and policy-based device DNS as well as the differences between the DNS resolver and the DNS forwarder daemons. The topic on DNS and name resolution DNS and name resolution has detail about how DNS works in the AppGate ZTNA system.
Checking DNS configuration
Check the following:
Site
Device DNS may rely on the Site configuration. Check that a DNS resolver and/or the DNS forwarder are correctly configured. It is normal to have one or both of these configured on the Site where the DNS servers are located.
If a user appears to have access to services on one Site and not on another then there might be a DNS resolver on the failing Site that might not be configured correctly.
DNS Policy
Entitlements (which set routes to protected networks) rely on DNS to resolve protected hosts; and only DNS requests that match a specified domain will be routed via the tunnel to the specified DNS server(s). Check that the hostnames used in entitlements have a suitable match domain specified in the DNS policy.
For Portal and mobile devices, you must have a default DNS server configured.
DNS Entitlement
The DNS server(s) specified above also needs an entitlement that allows DNS requests to be routed down the appropriate tunnel towards the required Gateway/Site hosting the DNS server(s). Check that there is an entitlement that provides access to the DNS server(s) on UDP port 53, which the device will be using to resolve (protected) hostnames.
Device DNS
Check the users device is not configured to use Private DNS, which might override any match domain settings set by AppGate ZTNA.
Diagnosing DNS issues
If there is an entitlement for the (protected) host but the user cannot access it, then it is likely to be related to DNS. On the Gateway check the DNS server IP addresses used in the appliance, the DNS resolver and the DNS forwarder. In these examples we are using the DNS Forwarder IPv4 address 100.127.20.30 (randomly generated when the DNS forwarder was configured) and forwarding DNS queries to a DNS server 10.10.10.10.
If the Gateway can't resolve the (protected) host then the device DNS will almost certainly not work either.
Check the Gateway's own DNS
On the Gateway do: dig github.com @10.10.10.10 where 10.10.10.10 is the IP address of the DNS server(s).
Then check the status for NOERROR. If you see NXDOMAIN then there is no DNS record for this (protected) host:
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14058
and check the IP under:
;; ANSWER SECTION:
github.com. 50 IN A 140.82.121.4
If the Gateway itself looks good then next we can check the DNS resolver (if using dns:// syntax) and/or DNS forwarder (if using *.domain.com syntax).
Check the DNS resolver and/or DNS forwarder on the Gateway
Name resolver
On the Gateway the user is connected to do: sudo cz-names status
Then check the list of resolutions:
dns://siteexample.com: ipv4, ipv6 (There will typically be an entry for each (protected) host (defined in the entitlement action) that signed-in users are allowed to access.)
For more information do: sudo cz-console -d cz-gonamed -p
This provides resolver process info for each configured resolver. This can include: Hits, Misses, Insertions, Evictions counts; the Update interval; Cache and Configuration info such as:
Status: healthy
Update interval: 1m0s
Match domains: [internal.company.com]
Zones transfers: false
AAAA lookups: true
Hostnames: 1
Check the cz-gonamed.log from the Gateway:
Look for the name being resolved and the ips that result.
DNS forwarder
On the Gateway the user is connected to do: sudo cz-console -d cz-dnsfwd -p
Then check for rules registered:
rules registered = 2 (The number should match the the number of match domains configured in the signed-in users DNS policy.)
And check the rules matched:
*.example.com: tun IPs (There will typically be an entry for each (protected) host (entitlement action) that each user (tun IP) has attempted to access.)
Check the cz-dnsfwd.log for:
INFO [wildcard][total-added=1][total-removed=0] Updating domain rules
INFO [wildcard][rule=*.bla.bla.com] Domain rule added
If the Gateways can't reveal where the issue lies, then you need to check the connecting device.
Checking the device's DNS
It is important to understand that DNS operates differently across the different operating systems. This can easily mean that one user is able to access a (protected) host while another (with the same policy/entitlements but on a different OS) might not get access - this will often come down to these DNS differences.
For instance, on macOS AppGate ZTNA uses the out-of-the-box match domain concept. Whereas in Windows AppGate ZTNA uses the Name Resolution Policy Table (NRPT) which Windows treats as a separate entity. Windows checks the NRPT before it sends a DNS query; if a DNS query matches an entry in the NRPT, it is handled according to settings in the table, otherwise it is processed using any configured DNS servers.
If the hostnames used in entitlements are not being resolved correctly, here are some things you can try:
Basic check of the device's DNS behavior
Cross-platform
If a (protected) host is not reachable (via an appropriate entitlement) then the first thing to do is to try using the IP address of the host instead of the DNS name.
The best cross-platform way to check DNS resolution is with Ping. Just ignore if the actual ping works or fails and just check the hostname's resolved IP address.
NOTE
The command
nslookup, it is not suited to checking AppGate ZTNA DNS configurations because of the advanced DNS configurations options used. On both macOS and Windowsnslookupdoesn't use the OS resolver and specifically it has no idea about AppGate ZTNA's DNS policies.
Checks to use when DNS has been configured using a DNS policy
Windows
The “normal” commands you may be used to are not appropriate when NRPT is used, so:
To check DNS resolution use the following DnsClient cmdlet command which allows various parameters that customize the DNS queries: Resolve-DnsName
To check NRPT settings use the Powershell command: Get-DnsClientNrptPolicy -Effective
In the response find the namespace you are looking for, and check the name server IP is correct.
This confirms that the NRPT entry has been added correctly; and all requests for .github.com will be sent to the DNS forwarder on 100.127.20.30.
NOTE
ipconfig /all will not show NRPT entries (from AppGate ZTNA) as Windows treats these differently from normal DNS server configurations.
macOS
To check DNS resolution the dscacheutil command can be used:
On the device open a command window and do: dscacheutil -q host -a name myapp.myco.com
name: myapp.myco.com
ipv6_address: fd00:ffff:a:1:2:3:4:5
name: myapp.myco.com
ip_address: 10.10.10.10
To check the DNS settings the scutil --dns command can be used.
In the response find the domain you are looking for, and check the nameserver IP is correct.
This confirms that the match domain entry has been added correctly; and all requests for .github.com will be sent to the DNS forwarder on 100.127.20.30.
Check the Client logs
Download the Client logs and unzip them. Find driver.log and open it. Search for DNS domain:
You should find a number of DNS domain entries - including one for the (protected) host. (These are a generic log entries on all operating systems.)
Info : DNS domain: github.com --> 100.127.20.30
And you should be able to find a number of “Added DNS policy entry” listed - including one for the (protected) host. (This is a Windows specific log entry relating to the NRPT record.)
Info : Added DNS policy entry: .github.com = 100.127.20.30
These confirm that the client knows about the DNS policy and has tried to add the DNS configuration.
Checking the Portal user's DNS
The portal includes a DNS proxy service which is designed to ensure each user gets the DNS policy/entitlement that is configured for them.
NOTE
The Portal required a default DNS to be configured.
There is a utility which can be used to view the live cache that ties the user to their client ID and their DNS settings. (Because this is a view of a live cache the results can be a bit unpredictable - so you may need to run this a few times to capture the information you are interested in.)
On the Portal, enter:
sudo cz-memcachedump
This will list: client id | username | status | DNS servers | DNS domains
Find the username of interest and check the DNS Servers are correct.
Checking Access Issues
Access to the appliances has already been addressed above; and at this point it is assumed that DNS is also working fine. So the here are a few things you can look at if the user still cannot access the (protected) hosts:
User's access rights
Only assigned policies will provision entitlements to the user. Make sure to use dns:// for the DNS resolver or *.domain.com for the DNS forwarder. Check the client for the assigned entitlements.
You can download the client logs from the client's About screen. Look in the log.log to:
...make sure the user has the required rights:
Info : Entitlements for 'Site': app1, app2, app3
...ensure any condition has been actioned successfully:
Info : Activate <name> remedy.
Info : Sending OTP pin to controller. RemedyProperties: 'ClaimSuffix <name>, ProviderId <ssid>, EntitlementName <Entitlement>, State: '', Timeout: '0'
Info : Successfully got a new claims token with OTP for <name>.
Routes
The client's About screen will give you the assigned IP address (of the tun adapter) - this will allow us to check which IP addresses are being routed through the tunnel.
On Windows do: netstat -rn
check the Network Destination is routed to the right Interface (tun IP).
On macOS do: ifconfig
find the network interface (Netif) name (such as utun2) related to the tun IP. Then do: netstat -rn
check the Destination is routed to the right Netif.
You can also download the client logs from the client's About screen. Look in the driver.log for the relevant entries:
On Windows look for: Info : [Site] route-add[37 10.10.10.10/32]: OK
On macOS look for: Info : [Site] Running 'route -n add -inet -host 10.10.10.10 -interface utun2'
NOTE
If using the DNS forwarder the routes are only written after access has been attempted.
Portal's web clients
Portal is quite similar to the normal client but to troubleshoot access issues you need to find the user's client. The client ID will therefore be needed before continuing with any troubleshooting.
It is included in memcachedump (client id), which was covered earlier in the DNS section; however it can also be obtained from the live session information (ClientID) or from the cz-webd.log (clientID).
Either, on the Portal enter: sudo nc -U /var/run/cz-webd/cz-webd-admin.socket then enter: sessions list
ShortID: vvvvvv ClientID: xxxx SrcIP: 1.2.3.4 Dn: CN=yyyyyy,CN=<username>,OU=zzzzzz SignedIn: true
Or, in the Portal's logs open the cz-webd.log file; search for CN=<user> and find the relevant (most recent?) log entry:
INFO [session][clientID=xxxx][cookie=wwww][dn=CN=yyyyyy,CN=<username>,OU=zzzzzz][loggedIn=true][srcIP=1.2.3.4] Client connected
From this log entry you can find the user's ClientID.
Once you have the client ID you can now go back to the logs and open the cz-webclient@xxxx.log file. This is the log record for that user's web client and contains similar information to the normal client logs.
If the client side of things look OK then the last thing to look at is the traffic through the Gateway to the protected host.
Gateway session
On the admin UI dashboard go to the session details for the user, select the Site where the (protected) host resides, and then look for the entitlement. This view is the actual live information from the Gateway the user is connected to at the moment. You can see the status of any conditions - green ones have been satisfied; red ones have not yet been met. The firewall rules shown will include the port/type and Hosts.
For more user session details, on the Gateway do: vpn-logcat -S today -u <username> list
This will list all <username>'s sessions today. Once you have the list then select the one of interest based in the time and do: vpn-logcat -S today -u <username> session <time>
This will now filter the logs for just <username>'s audit logs and display them sequentially. (See details below.)
For more specific information from the vpn daemon do: vpn-console --search-user <username>. (See details below.)
This will include details regarding Used Entitlements, Unused Entitlements, and URL Access.
If URL access (HTTP up Action type) is being used, then for more specific information from NGiNX do: curl -v http://127.0.0.1:9202/url_access/print_user_<tun IP>. The <tun IP> can be obtained from the client’s About screen or the user's session details - system claims. (See details below.)
App (Protected host) health
On the admin UI dashboard go to the session details for the user. In the global details at the top there is a value for the number of unreachable hosts the user has encountered. Select the Site where the problematic (protected) host resides and there is the value for the number of unreachable hosts the user has encountered on that Site.
If the user has been experiencing unreachable hosts, then more details can be found in App Health Reports.