Gateway and Portal troubleshooting commands

  • 2 minute read
Prev Next

Gateway

The users device's tunnels are terminated on the Gateways. Here cz-vpnd@x handles the majority of the traffic (NGiNX handles HTTP up), cz-sessiond controls the user's session; cz-gonamed or cz-dnsfwd resolves the resource names in Entitlements.

To list tun device

ip tuntap show

Capture traffic on a tun device, for example tun3

tcpdump -i tun3

View DNS forwarder details

cz-console -i

Shows the current status of the DNS Forwarder. This will include:

IPV4 stats
IPv6 Stats
IPC Channel stats
Wildcard rules Controller:

Display of wildcard rules for Google domain with registered and matched rules information.

Here it is possible to see each the matched domains listed. These are 'matched' being resolved and the resulting IP address.

View name resolver details

sudo nc -U /var/run/czd/cz-sessiond-admin.socket

Then enter named list

Command line output showing DNS information and timestamps for a Google domain.

Provides all the names subscribed to named along with the time it was sent, time it was updated, time it was received and results.

And there are some useful specific named commands:

sudo cz-names resolve > one time resolution of a name

sudo cz-names status > shows the current status (results) from named

sudo cz-console -d cz-gonamed -p > Shows details/information about the status of the resolvers and their resolved names

View user session details

vpn-logcat -S today -u Annie list > Annie's sessions today (with timestamps)

vpn-logcat -S today -u Annie session 2021-11-17T15:11:13.863611  > Specific session details from a given timestamp.

VPN session logs showing connection details and device creation timestamps.

View vpnd details

vpn-console [OPTIONS]

-h, --help > Print this help and exit

-s, --search-user <username> > Search for a user's sessions, across all VPN daemons

-r, --rules > Show session's IP rules

-c, --cert > Show session's certificate

-d, --decode-cert > Print session's certificate in human readable form

-i, --ipc > Show IPC information for all VPN daemons

-a, --all-sessions > Show all sessions of all VPN daemons

View NGiNX details (HTTP up Action type)

curl -v http://127.0.0.1:9202/url_access/print_user_<tun IP>

This will perform an http GET - the results will confirm NGiNX is operating as expected and will list the URIs in the [ ] for the specified for the user.

*   Trying 127.0.0.1.....
.....{"uris":[{"uri":"<hostname>:80","name":"<app name>","rule":"allow"},{"uri":"<hostname>:80/<subnet>","name":"<app name>","rule":"block"}],"dn":"CN=<deviceID>,CN=<username>,OU=local"}
* Connection #0 to host 127.0.0.1 left intact

Portal

The Portal uses the AppGate ZTNA Client under the covers. Since they are buried within an appliance, a set of troubleshooting tools are available to help diagnose any user access issues that might arise.

Webd is the daemon that controls the session usage within the Portal. To get to the webd admin console:

sudo nc -U /var/run/cz-webd/cz-webd-admin.socket

Then enter:

help > prints help

status > prints statistics for webd

sessions list > lists active sessions

session <SHORTID> info > prints info about specific session

session <SHORTID> remove > force removal of session

pool list > lists client pool usage

The sessions list will return entries like:

ShortID: uzczjd212f ClientID: 9000 SrcIP: 213.65.218.114 Dn: CN=566bd8c512384c70df714fe0f13ad7bd,CN=name.name,OU=AppGate-IdP LoggedIn: true

To see the specific client logs for a user's session.

journalctl -t cz-webclient@<ClientID>

ClientID can be obtained from the session list.

To check on the DNS settings being applied to each user use:

sudo cz-memcachedump

(Because this is a view of a live cache the results can be a bit unpredictable - so you may need to run this a few times to capture the information you are interested in.)

For a general overview of the Portal's Clients and the related memory consumption use:

sudo cz-clients status

Related articles