Troubleshooting network connectivity

Prev Next

If users have difficulty connecting to AppGate ZTNA Controllers or Gateways, check the following three areas in order:

  1. SPA traffic

  2. DNS resolution

  3. TLS connections

For Controller connectivity issues, the client logs will include the following entry:

Info : REST connection could not be established. 'Connection failed.'. Will try the next IP address.


SPA issues

SPA can cause initial connectivity problems. Once identified and resolved, SPA traffic is handled like all other TLS-based network traffic. If you are not familiar with SPA, see the SPA overview page for details on its operation and benefits.

To diagnose SPA problems, open cz-proxyd.log. This is the front-door log for all AppGate ZTNA appliances and records all SPA packets. If you know the IP address of the connecting party, search for it.

A TLS connection using SPA always sends two UDP packets followed by one TLS packet as part of the SPA process, regardless of the SPA mode. The following sections describe the issues to look for in the logs.

UDP packet loss

If UDP-TCP SPA is in use, both of the following lines must appear in the logs. If you are using a QUIC tunnel, only the DNS line appears as QUIC uses UDP port 443 and there fore cannot use SPA-DTLS.

INFO [spa] SPA-DTLS from 14.137.17.12:51114 has been authorized by using the key with name "keyname"

INFO [spa] SPA-DNS from 14.137.17.12:52423 has been authorized by using the key with name "keyname"

Only one packet needs to reach the appliance, but if one is missing, a firewall rule change is likely required to allow both through.

When a valid SPA key is used, the system updates iptables to allow TCP traffic from that address, which permits the subsequent SPA-TLS packet to reach proxyd.

TCP packet loss

After the two UDP packets above—or when TCP-SPA is in use—a successful new TCP connection produces the following log entries:

INFO [server] New connection from 14.137.17.12:49538
INFO [spa] SPA-TLS from 14.137.17.12:49538 has been authorized by using the key with name "keyname"
INFO [server] Connection established with /run/czd/cz-nginx-main-client.socket

If the UDP packets appear but these entries are missing, the cause is likely one of the following.

  • UDP differential packet routing. Some mobile internet service providers (ISPs) route valid UDP SPA packets from a different IP address than the one used by the TCP connection. When this happens, the system updates iptables to allow TCP from the UDP source address, so the SPA-TLS packet from the TCP address is blocked and does not reach proxyd.

  • Firewall filtering of TLS. Some application-aware firewalls apply strict TLS filtering that detects the SPA extensions in TLS client hello packets and drops them. A firewall rule change is required. If allowing all TLS traffic does not resolve the issue, try allowing TCP 443. Also check for and remove any application-specific filters.

Unauthorized SPA packet

An unauthorized connection produces log entries similar to the following:

INFO [server] New connection from 64.22.1.39:43631
ERROR [spa] SPA message has expired! Current time: 2021-10-21 13:04:56 +0000 UTC SPA message timestamp: 2021-10-20 21:17:24 +0000 UTC Difference: 15h:47m:32s Max allowed difference: 3600 seconds
ERROR [spa] SPA-TLS from 64.22.1.39:43631 failed to get authorized. Dropping!

SPA packets include a time element designed to prevent replay attacks. If the connecting device has an incorrectly configured NTP server, the time offset may be large enough to prevent the packet from being authorized. Check the current time and NTP settings on the connecting device.

Invalid SPA packet

When a connection attempt reaches an open TCP port without a valid SPA packet—for example, when UDP-TCP SPA is not in use—the logs show:

INFO [server] New connection from 80.216.53.180:44462
WARNING [server] 80.216.53.180:44462 has closed the connection

The AppGate ZTNA system treats this as an invalid SPA packet and does not establish a TLS connection.


DNS resolution issues

DNS resolution failures can prevent users from connecting to Controllers and Gateways, and may also cause issues after the tunnel is established. The following are typical DNS-related issues.

No DNS record for the profile DNS name

When the first Controller is created, a Profile DNS Name is assigned and shared across all Controllers. A DNS record must exist for this name with one A record for each Controller in the Collective. If this record is missing, clients cannot connect.

Controller hostname matches an internal search domain

The following sequence describes how an internal DNS search domain can prevent clients from reconnecting to the Controller after a session is established:

  1. When the client starts, it resolves appgate.b.com using local DNS, which succeeds.

  2. The client connects to the Controller or Gateway and receives the search domain b.com and its associated internal DNS servers, as configured in the identity provider.

  3. The client configures the operating system to route requests for *.b.com to the internal DNS servers, which are accessed through the VPN tunnel.

  4. Time passes and the client needs to contact the Controller again—for example, to renew a token.

  5. The cached DNS entry for appgate.b.com has expired. The client performs a new DNS lookup, which now goes to the internal DNS servers.

  6. Each DNS server is tried in turn with a timeout of approximately two seconds. The lookup eventually fails with: Warning: Lookup of 'appgate.b.com' failed: Timeout when looking up 'appgate.b.com'.

Internal DNS returns an internal IP address for the Controller

Once connected, the client caches the external IP addresses for Controllers and Gateways to reduce the risk of this issue. However, if the cache expires and the device is configured to query internal DNS servers first, those servers may return an internal IP address for appgate.b.com. A client connecting from outside the network cannot route to an internal IP address and loses the ability to communicate with the Controller through the Gateway.


TLS issues (man-in-the-middle)

AppGate ZTNA uses mutual TLS (mTLS) and does not accept man-in-the-middle (MITM) interception. If a firewall or web gateway intercepts traffic between the client and the appliances, the TLS connection fails and the user sees an "Invalid certificate" warning.

Sign-in page displaying an invalid certificate warning.

To identify the source of the interception, download and extract the client logs, then check the following files.

log.log

Search for unknown CA. Look for ERR_CONNECT_CA_CERT_ERROR, which confirms a certificate mismatch:

Info : Trying to connect 'server.company.com/1.2.3.4' with SPA key name '<name>'.
Info : REST  (server.company.com:443) H0 [[12]] : Connection failed. ERR_CONNECT_CA_CERT_ERROR - [server.company.com - 0]: Server reported: unknown CA
Info : SSL certificate of controller 'server.company.com' was not accepted. Trying the next CA certificate stored in the system.

driver.log

Search for unknown CA. A few lines above this entry, look for the server certificate details:

appgatedriver[11164]: [Site] Server certificate DN: CN=whatever.com
appgatedriver[11164]: [Site] Server certificate issuer: CN=R3,O=Let's Encrypt,C=US
appgatedriver[11164]: [Site] Server certificate SAN.DNS: api.whatever.com
appgatedriver[11164]: [Site] Connection closed: end of file - Server reported: unknown CA
appgatedriver[11164]: [Site] [<Gateway>] Closing (Disconnected by gateway - Server reported: unknown CA)

The CN value identifies the device intercepting the TLS traffic. Once identified, configure that device to allow all TCP 443 traffic without interception.