Troubleshooting user and device access

Prev Next

User access in AppGate ZTNA follows a six-layer model, so there are several points in the connection flow where issues can prevent access from working as expected.

Before troubleshooting, verify that the client is correctly installed and working. See Installing and updating clients.

For full details on setting up access to protected hosts using entitlements, see User and device access.There are also Client > Host and Client > Client checklists covering all network-related configuration required for users to reach protected hosts. Work through these checklists first, as they may resolve the issue without requiring the steps in this section.

Connected users appear in Usage > Active Sessions, where you can examine a specific user. The Gateway and client both provide statistics that help identify connectivity issues, including connection times, connection retry counters, round-trip time (RTT), and DNS failure counters.

If the steps in this section do not help, see the following sections:


Connectivity checklists

Client to Controller

The client must be able to send traffic to the Controller, and the Controller must accept it. This requires the client profile to include a resolvable hostname. Check each of the following:

  • The hostname: The client connects to the Profile DNS name generated when the first Controller ran cz-setup. Verify that the client profile in use is the one you expect.

    Client Connections form for with the field for entering Profile DNS Name.

  • Port: The client connects on the default port shown in the appliance’s System TLS Connection settings. Verify that the port is correct and that inbound traffic is allowed on that port.

  • DNS resolution: Verify that the client can resolve the Profile DNS name to the correct Controller IP addresses using public DNS. Check from the client that the name resolves to a valid IP address.

  • Sending traffic to the Controller: Firewall rules sometimes block traffic from the Internet to the Controller. This can be compounded because of the use of single-packet authorization (SPA). Check the SPA settings and verify that the appropriate ports are open for TCP and/or UDP. See the SPA issues section for more information.

    Options for SPA key checks before allowing connections, highlighting UDP and TCP settings.

  • Wrong SPA key: If the client is using a deleted SPA key, the connection fails. Verify that the client has been seeded with a valid SPA key using the client profile. Check the client logs and session details to confirm which SPA key is in use.

  • Incorrect time: SPA packets are valid only within a narrow time window. If the time is incorrect on the client or Controller, the connection fails. Check the proxyd logs on the Controller for an entry similar to the following:
    [2020-03-18T16:33:21.093Z] [M 7fbe0420dd40] SPA data have expired! 25 seconds above the time-window: 3600. current time: 1584549201, SPA message's timestamp: 1584552826. Check the dashboard for NTP warnings. Verify the time on the client.

  • Allowed sources: A restriction may be set on the IP addresses permitted to send traffic to the System TLS port. Check Allowed Sources in the System TLS Connection settings and verify that the IP address presented by the client is allowed.

  • CA certificate mismatch: A certificate mismatch most commonly occurs when the client misses a CA update, which can happen if the client has been offline for an extended period.

    During the transitional phase, the next CA certificate is distributed to every client that connects. Sufficient time should be allowed to ensure all clients have connected at least once; a client will only pick up the next CA certificate at sign-in or when the claims token is renewed.
    Appliances in the Collective use the same process, but this should be almost immediate. New client profiles generated after this time will include the next CA certificate.

    When a certificate mismatch occurs between the Controller and the client, the user sees an "Invalid certificate" message. Identify the cause of the mismatch to confirm it is not due to a man-in-the-middle (MITM) attack. If the mismatch is caused by a known change, resolve it by deleting the certificate from the user's machine, or by uninstalling and reinstalling the AppGate ZTNA client.

    NOTE

    Issuing a new client profile does not resolve a certificate mismatch if you are only replacing a unique profile.

Clients load-balance across available Controllers by selecting one IP address at random. Once a client establishes a connection with a Controller, it retains that IP address as its preferred address in memory until the address becomes unresponsive or the client restarts. This behavior improves reliability for interactions that involve multiple connections, such as certain multi-factor authentication (MFA) methods.

Client to Gateways

The client must be able to send traffic to the Gateway, and the Gateway must accept it. This requires the entitlement token to include a resolvable hostname. The flow is similar to the Controller flow above, but with some differences. When the client cannot connect to one or more Sites, those Sites appear grayed out.

Appgate SDP Client showing connected sites and their status information.

Check each of the following:

  • The hostname: The client connects to the client hostname or IP address set in Secure Tunnel Settings, using the default ports shown. Verify that the hostname in Secure Tunnel Settings matches your DNS record.

  • DNS resolution: If the client hostname or IP address is a fully qualified domain name (FQDN), verify that the client can resolve the name to a valid IP address. This resolution typically uses public DNS. Check from the client that the name resolves to a valid IP address.

  • Sending traffic to the Gateway: Firewall rules may block traffic from the internet to the Gateway.

    • For each Site you, you can select TLS or QUIC as the protocol. If using TLS, verify that TCP traffic is allowed. If using QUIC, verify that UDP traffic is allowed.

    • System-generated keys are used for SPA, so SPA key errors do not apply here. However, firewall rules may still block traffic from the internet to the Gateway. Check the SPA settings and verify that the appropriate ports are open for TCP and/or UDP. See the SPA issues section for more information.

  • Incorrect time: SPA packets are valid only within a narrow time window. If the connection to the Controller succeeded but the connection to the Gateway fails, the Gateway time may be incorrect. Check the proxyd logs on the Gateway for an entry similar to the following: [2020-03-18T16:33:21.093Z] [M 7fbe0420dd40] SPA data have expired! 25 seconds above the time-window: 3600. current time: 1584549201, SPA message's timestamp: 1584552826. Check the dashboard for NTP warnings.

  • Allowed sources: A restriction may be set on the IP addresses permitted to send traffic to the TLS port (default 443). Check Allowed Sources in the System TLS Connection settings and verify that the IP address presented by the client is allowed.


Appliance and client logs

If users are experiencing access issues, check the appliance audit logs in the LogServer. Filter logs by traffic source using distinguished_name_user, add a filter on traffic destination using hostname (log_source on LogServer, OpenSearch, or Elasticsearch), then show event_type.

Search for the following event types:

  • tunnel_connected and tunnel_established — indicate a connection from the client

  • ip_access — shows traffic flowing through the tunnel; the action field shows what is happening to the user's traffic (allow, drop, reject, or block_report)

If traffic is being dropped, the drop-reason field identifies the cause. A value of No matching rule found indicates the firewall is operating as expected.

If you do not see any traffic from the user in the audit logs, check the appliance daemon logs. Download logs from the Gateway appliance the user is trying to connect to. Extract the archive and check the following files in logs_by_daemon:

  • cz-proxyd.log — the front-door log; confirms whether traffic is reaching the appliance

  • cz-vpnd@x.log — the firewall daemon log; confirms whether traffic is passing through the appliance. There is one file per vCPU present in the Gateway.

  • cz-gonamed.log — the resolver log

  • cz-dnsfwd.log — the DNS forwarder log

If the user does not appear in the appliance logs, download the client logs. Log levels can also be increased for each operating system if more detail is required. See Client deployment and management for more information.