Gateway and Portal troubleshooting commands

Prev Next

Gateway

User device tunnels terminate on the Gateways. The following daemons handle traffic and session management on each Gateway:

  • cz-vpnd@x handles the majority of traffic

  • NGINX handles HTTP up actions

  • cz-sessiond controls the user's session

  • cz-gonamed or cz-dnsfwd resolves the resource names defined in entitlements

List tun devices

To list all tun devices, run:

ip tuntap show

Capture traffic on a tun device

To capture traffic on a tun device—for example, tun3—run:

tcp dump -i tun3

View DNS forwarder details

To view the current status of the DNS forwarder, run:

cz-console -i

The output includes the following:

  • IPv4 stats

  • IPv6 stats

  • IPC channel stats

  • Wildcard rules

Under Wildcard rules, each matched domain is listed along with the resolved IP address.

Display of wildcard rules for Google domain with registered and matched rules information.

View name resolver details

To view all names subscribed to named, along with sent, updated, and received timestamps and results, run the following commands in sequence.

First, open the admin socket:

udo nc -U /var/run/czd/cz-sessiond-admin.socket

The enter:

named list

The following additional commands are also available for working with the name resolver:

Command

Description

sudo cz-names resolve

Performs a one-time resolution of a name.

sudo cz-names status

Shows the current status and results from named

sudo cz-console -d cz-gonamed -p

Shows details about the status of the resolvers and their resolved names.

View user session details

To view session information for a specific user, use vpn-logcat. The following examples use the username Annie:

To list all of Annie's sessions today with timestamps:

vpn-logcat -S today -u Annie list

To view details for a specific session, run:

vpn-logcat -S today -u Annie session 2021-11-17T15:11:13.863611

Replace the timestamp with the value from the session list.

View VPN daemon details

The vpn-console command provides session and daemon information. The following options are available:

Option

Description

-h, --help

Prints help and exits

-s, --search-user <username>

Searches for a user’s sessions across all VPN daemons

-r, --rules

Shows the session’s IP rules

-c, --cert

Shows the session’s certificate

-d, --decode-cert

Prints the session’s certificate in human-readable form

-i, --ipc

Shows IPC information for all VPN daemons

-a, --all-sessions

Shows all sessions of all VPN daemons

View NGINX details (HTTP up action type)

To view NGINX details for a specific user, run:

curl -v http://127.0.0.1:9202/url_access/print_user_<tun IP>

Replace <tun IP> with the tun IP address from the client’s About screen or the user's session details.

This performs an HTTP GET request. The response confirms that NGINX is operating as expected and lists the URIs configured for that user:

*   Trying 127.0.0.1.....
.....{"uris":[{"uri":"<hostname>:80","name":"<app name>","rule":"allow"},{"uri":"<hostname>:80/<subnet>","name":"<app name>","rule":"block"}],"dn":"CN=<deviceID>,CN=<username>,OU=local"}
* Connection #0 to host 127.0.0.1 left intact

Portal

The Portal uses the AppGate ZTNA client internally. A set of troubleshooting tools is available to help diagnose user access issues.

cz-webd is the daemon that controls session usage within the Portal. To access the webd admin console, run:

sudo nc -U /var/run/cz-webd/cz-webd-admin.socket

The following commands are available in the admin console:

Command

Description

help

Prints help

status

Prints statistics for webd

sessions list

Lists active sessions

session <SHORTID> info

Prints information about a specific session

session <SHORTID> remove

Forces removal of a session

pool list

Lists client pool usage

The sessions list command returns entries in the following format:

ShortID: uzczjd212f ClientID: 9000 SrcIP: 213.65.218.114 Dn: CN=566bd8c512384c70df714fe0f13ad7bd,CN=name.name,OU=AppGate-IdP LoggedIn: true

View client logs for a user session

To view the client logs for a specific user session, run:

journalctl -t cz-webclient@<ClientID>

Obtain the CliendID value from the sessions list output.

Check DNS settings for Portal users

To view the DNS settings applied to each user, run:

sudo cz-memcachedump

NOTE

This command reads a live cache. Run it more than once if needed to capture the information you are looking for.

View Portal client status and memory usage

To view a general overview of Portal clients and their memory consumption, run:

sudo cz-clients status