Appliance commands reference

Prev Next

This topic covers commands for appliance troubleshooting, including how to view logs and daemon information. AppGate ZTNA appliances run a customized version of Ubuntu 22.04. The commands below are based on standard Linux and include AppGate ZTNA-specific information where relevant.

For the healthcheck messages these commands help you investigate, see Appliance health checks and diagnostics. For user and device connectivity issues, particularly for Gateways, see Troubleshooting user and device access.


Run commands

Selecting Run Commands from an appliance opens the Remote Commands window. Eight remote commands are available to run directly on an appliance, reducing the need to SSH to remote machines for basic diagnostics:

  • address show

  • dig

  • ip route show

  • netcat

  • ntpq

  • ping

  • tcpdump

  • traceroute

Most commands include a Timeout field that accepts a value in seconds.


Daemon log commands (SSH)

journalctl

Use the following journalctl commands to view daemon logs.

To follow live logs:

journalctl -f

To show logs for a specific service:

journalctl -u cz-configd
journalctl -u cz-vpnd@0 # vpnd instance 0
journalctl -u "cz-vpnd@*" # all vpnd instances
journalctl -u appgatedriver@<client name> # Connector client tun driver
journalctl -u appgateservice@<client name> # Connector client service
journalctl -u cz-nginx@admin # admin and API interface
journalctl -u cz-nginx@urlaccess # HTTP up action type
journalctl -u cz-nginx@portal # Portal

To filter by severity level:

journalctl -p warning

To show logs in reverse order:

journalctl -r

To show logs since the last boot:

journalctl -b

To show logs in a specific time range:

journalctl --since="2017-06-01 12:17:16" --until="2017-06-02"

You can combine all flags.

syslog

journalctl is sufficient for viewing logs in most cases. If the binary logs are corrupted, fall back to /var/log/syslog. Earlier versions of the product used /var/log/upstart.

Saving logs

AppGate ZTNA produces two types of log records: daemon logs and audit logs. Daemon logs record the workings of the AppGate ZTNA system. Audit logs record actions performed by the system.

Logs are saved automatically on each appliance. Download logs from Appliances (System > Appliances) for local examination, or copy them to another machine using scp or sftp.

Debug log level

The daemon log level setting controls which system events the appliance stores in its debug log. For troubleshooting, run the appliance in DEBUG mode. Once troubleshooting is complete, reset the log level to a lower setting to maintain optimal system performance.

For instructions on changing debug log levels, see Audit logs.


Troubleshooting commands (SSH)

Use the SSH command line to run the following troubleshooting commands.

View the current version:

cat /usr/share/cz-image/version

View the current appliance state:

cat /mnt/state/last-state

View uptime and system load:

uptime

Reboot the appliance:

sudo reboot

Some clients may need to reconnect.

Restart appliance services without a full reboot:

sudo service cz-configd restart

Restart an arbitrary daemon:

sudo service <daemon> restart

Bypass root:

This requires access to the GRUB menu. Reboot the appliance, then press e to edit the GRUB menu. Append cz-login=root to the kernel line, then press Ctrl-x as shown in the GRUB menu instructions.
Image

Collect appliance diagnostics:

sudo cz-system-info [--full]

This command collects a full set of appliance diagnostic information, including license usage, and saves it to /tmp/cz-system-info.txt.gz. The --full option adds information about the state of certain processes. The file is owned by root. To make the file owned by the cz user, which may make it easier to transfer, run:

sudo cz-system-info; sudo chmod a+r /tmp/system-info.txt.gz

To download the file using SCP:

scp -i ~/keys/mykey cz@controller.myco.com:/tmp/cz-system-info.txt.gz ~/Desktop/cz-system-info.txt.gz

Remove a core dump:

Core dumps are stored in /mnt/data/core. Remove files from that directory with sudo rm to remove the core dump and the associated dashboard warning.

Update the GeoIP database immediately:

sudo systemctl start cz-geoip --no-block --show-transaction

View available and used memory:

free -mh

View processes by resource usage:

htop

View running processes:

ps -aux
ps -aux | grep <pid>
ps -ef

View current system firewall rules:

iptables -L -v

For IPv6:

ip6tables -L -v

View network interface addresses:

ip addr show
ip link show

For IPv6:

ip -6 addr show
ip -6 link show

View routes:

ip route show

For IPv6:

ip -6 route show

View configuration files:

Each AppGate ZTNA daemon stores its configuration file under /etc/cz-<daemon-name>. For example:

  • sessiond: /etc/cz-sessiond

  • NGINX: /etc/nginx

  • Rsyslog: /etc/rsyslog.conf

The current combined system configuration is stored under /mnt/state/config/current and contains the following files:

  • local.json: the appliance local configuration

  • remote.json: the appliance remote configuration

To view these files, run jql (local) or jqr (remote) from anywhere in the file system. The previous combined system configuration is stored under /mnt/state/config/previous.


Advanced BDR commands

WARNING

These commands are intended to help recover from failures when adding or removing a Controller. Contact support before using any of these commands.

Replace <option> in sudo cz-config <option> with the relevant BDR command:

Command

Description

bdr status [--show-parted-nodes --exclude-raft --JSON --table-fmt]

Displays human-readable BDR status for all Controller databases. Options: --show-parted-nodes includes already-parted nodes; --exclude-raft excludes RAFT status; --json outputs JSON; --table-fmt sets table format (use psql for terminals such as PuTTY).

bdr force-single-controller-ready

Forces the appliance to single_controller_ready state. Use with caution.

bdr force-appliance-ready

Forces the appliance to appliance_ready state. Use with caution.

bdr clear-barrier

Clears all BDR barriers. Use with caution.

bdr remove-node-record REMOVE_NODE_RECORD

Forcefully removes a node from BDR on the current node.

bdr update-bdr-group

Takes the current BDR leave barrier in the name of a dead node.

bdr enable-ip-allocation

Enables IP allocation for the current node.

bdr disable-ip-allocation

Disables IP allocation for the current node.

bdr repartition-ip-allocations

Repartitions IP allocations to match current Controllers.

bdr resolve-conflicts [--run-local-node --keep-conflict-history]

Resolves BDR conflicts. --run-local-node runs the cleanup query on this node only (the default is all nodes); --keep-conflict-history retains conflict history.

bdr --help

Displays help.

For scenario-based guidance on using these commands, see Controller maintenance and troubleshooting.