Appliance healthchecks and diagnostics

Prev Next

The Collective performs about 50 Site, appliance, and functional healthchecks. Results appear on the dashboard in the Sites widget and the Appliances widget. From either widget, go to Appliance Health Details to view warnings or errors.

This topic describes the healthcheck messages you may see and the action to take for each one. For SSH commands and other diagnostic tools, see Appliance commands reference.


Errors

The following table describes error-level healthcheck messages and the action to take when your Collective reports one.

Source

Urgency

Message

Action

Appliance

High

Current Controller cannot reach the appliance on appliance-hostname:443

Verify the appliance is running. Verify that the Controller hostnames are resolvable on the appliance and that port 443 is open to the Controllers. Verify the time on the appliance. SSH to the appliance and run nc -zv controller-hostname 443. This command returns succeeded if the Collective is in TCP-SPA mode. If it does not, check the network firewall for man-in-the-middle interference.

Appliance

High

I/O Stalled

The disk I/O stalled for several seconds, indicating either an availability issue or a capacity problem with the underlying storage system. Verify hardware diagnostics or check your hypervisor stack for storage issues.

Appliance

Low

I/O Error

The underlying storage system of the hardware or hypervisor is reporting an error. Verify that storage is working correctly, run storage diagnostics, and check for read or write issues on the underlying disk. I/O errors can lead to data corruption.

Appliance

Medium

GeoIP database missing

The appliance cannot download GeoIP data from https://bin.appgate-sdp.com or https://updates.maxmind.com. Verify that the appliance has internet access and that DNS is working correctly. If GeoIP data is not required, or external server connections are not permitted, disable GeoIP in Global Settings (Settings > Global Settings).

Appliance

Medium

Failed to read NTP status

NTP status cannot be verified. On the appliance, run sudo ntpq -np and check for errors. This issue is most likely caused by a DNS or connectivity problem that prevents the appliance from reaching the configured NTP servers.

Appliance

High

Failed to perform healthcheck for <appliance name>

The healthcheck service is not running on this appliance. Check cz-configd logs for details.

Appliance

Medium

Not connected to any Controller

The appliance cannot reach any Controllers in the Collective. Verify that the appliance can reach Controller TCP port 443 (default). If UDP-SPA is enabled, verify that the appliance can connect to UDP ports 53 and 443 on the Controllers. Verify that the time is synchronized across all appliances.

Appliance

High

Customization error

The appliance has a broken customization script. Download the appliance logs and check logs_by_daemon/cz-customization.log.

Appliance

High

Stuck initializing cloud instance

The appliance is expecting cloud-init information but is not receiving it. Verify your network settings and confirm with your cloud provider that it is sending cloud-init data. Verify that DHCP settings for DNS and the default gateway are enabled, as most platforms require these to receive cloud-init information.

Appliance

Medium

The following services are not running

One or more daemons that are expected to be running have not started. Sign in to the appliance and check the daemon status: sudo systemctl status <daemon name>.

Appliance

Medium

High volume usage <n> [X%]

The specified volume is more than 90 percent full. Check what is consuming disk space and remove unnecessary files, such as old core dumps.

Controller

High

Unable to connect dbd instance

The Controller cannot reach the database daemon. Check cz-dbs for status and contact support.

Controller

High

Unexpected state for running Controller

Contact support.

Controller

High

IP Pool <n> has X IPs allocated out of Y (90 percent or higher usage)

The Controller is running low on IPs from the IP pool. Go to IP Pools (Identity > IP Pools) and compare currently used IPs against the total pool size. If used IPs are close to the total size, either reduce the lease time to free up IPs or add additional ranges. When adding ranges, verify that they are routed correctly for each Site not using SNAT. If used IPs are at least 50 percent lower than the total size, users are likely reaching only one Controller; resolve the connectivity issue so the new Controller can assign the unused IPs.

Gateway

High

Failed to query cz-sessiond for status

The cz-sessiond daemon is not working correctly. Restart it with sudo systemctl restart cz-sessiond.

Gateway

High

Very high number of active connections

The connection tracking table has reached 95 percent capacity. At 100 percent, users may experience dropped application sessions. Check conntrack settings with `sudo sysctl -a

Portal

High

Failed to query cz-nginx@urlaccess for status

Run sudo systemctl status cz-nginx@urlaccess to check that the service started correctly.

Portal

High

cz-nginx@urlaccess: Shared memory size is not enough to save all the HTTP up action objects and the auxiliary data

Add more memory or reduce the number of HTTP up actions.

LogServer

Medium

OpenSearch is down or starting up

Run sudo systemctl status cz-opensearch to check that the service started correctly.

LogServer/LogForwarder

Medium

cz-logd: Unable to connect to Elasticsearch

LogServer or LogForwarder cannot communicate with Elasticsearch. Check whether Elasticsearch is down.

LogServer/LogForwarder

Medium

cz-logd: Unable to prepare POST request for inserting data into Elasticsearch

LogServer or LogForwarder cannot post inserts into Elasticsearch. Check connectivity to the HTTP port of Elasticsearch and verify that the Elasticsearch configuration matches the LogForwarder configuration.

LogServer/LogForwarder

Medium

cz-logd: Unable to create health request for Elasticsearch

LogForwarder or LogServer cannot query health information from Elasticsearch. Check Elasticsearch connectivity and configuration. For LogServer, also verify that the Elasticsearch service is running.

LogServer/LogForwarder

Medium

cz-logd: Unable to create index into elasticsearch, got status: X

LogServer or LogForwarder cannot create indexes in Elasticsearch. Check the Elasticsearch configuration.

LogServer/LogForwarder

Medium

cz-logd: Elasticsearch status is not green/yellow

The Elasticsearch status is not green. Check the Elasticsearch server status and restore it to green or yellow.

LogForwarder

Medium

cz-logd: Unable to get stream, does it exist?, streamname: X, details: Y

LogForwarder is configured with a stream that does not exist in AWS. Verify the configured stream name or create it in AWS.

LogForwarder

Medium

cz-logd: Unable to get delivery stream, does it exist?, streamname: X, details: Y

LogForwarder cannot get the delivery stream. Check IAM roles for the Kinesis output.

LogForwarder

Medium

cz-logd: Could not compile filters, X

LogForwarder has a filter configured that is not compiling. Check the filter configuration for the LogForwarder.

LogForwarder

Medium

cz-logd: No credentials provided

LogForwarder Kinesis output has no credentials or invalid credentials. Check the Kinesis LogForwarder configuration.

LogForwarder

Medium

cz-logd: Unable to get AWS region, details: X

LogForwarder cannot retrieve information about an AWS region. Verify that the correct region is configured for the Kinesis output.

LogForwarder

Medium

cz-logd: Unable to create AWS session, details: X

LogForwarder cannot create an AWS session using the AWS SDK. Check AWS credentials and region configuration.


Warnings

The following table describes warning-level healthcheck messages and the action to take when your Collective reports one.

Source

Urgency

Message

Action

Appliance

Low

GeoIP database was last updated X days ago

The appliance did not receive the latest GeoIP data from https://bin.appgate-sdp.com or https://updates.maxmind.com. Verify that the appliance has internet access and that DNS is working correctly. To force an update manually, run sudo /etc/cron.daily/geoIpDbUpdate --force.

Appliance

Medium

High volume usage <n> [X%]

The specified volume is more than 75 percent full. Check what is consuming disk space and remove unnecessary files, such as old core dumps.

Appliance

Medium

Certificate with subject <n> for <appliance name> has expired

An appliance certificate has expired. The appliance stops accepting connections when this occurs. Certificates renew automatically from version 6.1 onward, so this warning appears only if the appliance was offline during renewal.

Appliance

Medium

Certificate with subject <n> for <appliance name> is expiring

An appliance certificate expires within 30 days. Select the renew certificate option in Appliances (System > Appliances). Renewing the certificate restarts all services on this appliance.

Appliance

Medium

The following services have debug logs enabled

Running with debug logs enabled may affect performance. Switch back to normal log levels once troubleshooting is complete.

Appliance

Low

Configuration from Controller is incompatible with this appliance

The configuration from the Controller does not match the configuration format of this appliance. This is likely caused by a version incompatibility.

Appliance

Low

cz-ffwd: Unable to connect to X@X (X)

The appliance cannot establish a WebSocket connection to the LogServer or LogForwarder. Check connectivity between appliances (default TCP port 443, UDP ports 53 and 443).

Appliance

Low

This system has a CD drive attached

The appliance is running on VMware and still has a CD drive attached. Remove the attached CD drive from the virtual machine in the VMware console.

Controller

High

There are more X than your license allows

The number of users exceeds the licensed limit. Access is granted to the first set of licensed users only. Contact support or sales to update your license.

Controller

High

IP Pool <n> is too small to be utilized by this Controller

The assigned IP pool has fewer IPs than the number of Controllers. For example, a /30 pool with six Controllers produces this error. Assign a larger IP pool.

Controller

High

Controller appliance certificate does not include the client profile DNS name <n>

The client profile DNS name is not present as a subject alternative name (SAN) in the appliance certificate. Renew the certificate from the Appliances page (System > Appliances).

Controller

Medium

Optimizer is not active because this Collective is not registered to ZTP.

Register the Collective to ZTP.

Controller

Medium

Optimizer ZTP service is not available.

Check the status of your ZTP service.

Controller

Medium

Optimizer is not active because there is no active LogServer or LogForwarder in this Collective.

Add a LogServer or LogForwarder to the Collective.

Controller

Medium

X of the user licenses are in use

You are approaching the user license limit. Contact support or sales to update your license count.

Controller

Medium

X of the Portal licenses are in use

You are approaching the Portal license limit. Contact support or sales to update your license count.

Controller

Medium

X of the service licenses are in use

You are approaching the service license limit. Contact support or sales to update your license count.

Controller

Medium

Controller is running in maintenance mode

The Controller entered maintenance mode during an upgrade. If the upgrade was cancelled and the node remains in maintenance mode, run sudo cz-config set -j Controller/maintenance false to exit maintenance mode, only after confirming the upgrade has been cancelled.

Controller

Medium

Database node not replicating

This Controller cannot replicate the database with another Controller. Check connectivity between the two Controllers; bidirectional connectivity is required.

Controller

Medium

IP Pool <n> has X IPs allocated out of Y (75 to 90 percent usage)

The Controller is running low on IPs from the IP pool. Go to IP Pools (Identity > IP Pools) and compare currently used IPs against the total pool size. If used IPs are close to the total size, either reduce the lease time to free up IPs or add additional ranges. When adding ranges, verify that they are routed correctly for each Site not using SNAT. If used IPs are at least 50 percent lower than the total size, users are likely reaching only one Controller; resolve the connectivity issue so the new Controller can assign the unused IPs.

Controller

Low

BDR conflict

Different Controllers have conflicting versions of data, likely caused by a temporary network connectivity issue. Most conflicts resolve automatically by accepting the latest record update. To resolve remaining conflicts, run sudo cz-config bdr resolve-conflicts. If this warning continues to appear, contact support.

Controller

Low

The following entitlements are using the deprecated Risk Based Access feature

Migrate these entitlements to Condition Based Access, which provides the same functionality using conditions with risk score criteria.

LogForwarder

Medium

cz-logd: Not connected to X

A LogForwarder output named X is not connected. Check connectivity from the LogForwarder to the log destination.

LogForwarder

Medium

cz-logd: Unable to perform log-retention

The LogForwarder cannot remove indexes from Elasticsearch. Check Elasticsearch connectivity and configuration.

LogForwarder

Medium

cz-logd: Failed to send logs to X

Connectivity for an HTTP-based output is not working. Check connectivity from the LogForwarder to output destination X.

LogForwarder

Medium

cz-logd: Kinesis was unable to handle all records

The Kinesis output is being throttled. Configure additional shards in AWS.

LogForwarder

Medium

cz-logd: Kinesis error codes: X

AWS is returning an error code for the Kinesis output. These errors are typically IAM-related. Check and correct the AWS IAM configuration.

LogForwarder

Medium

cz-logd: Firehose was unable to handle all records

The Kinesis Firehose output is being throttled. Configure more resources on the AWS side to handle the load.

LogForwarder

Medium

cz-logd: unable to connect to LogForwarding destination

The LogForwarder has a TCP-based output configured that cannot connect. Check connectivity to the configured output.

LogForwarder

Medium

cz-logd: TCP output is slow

The rate of log generation exceeds the rate of log transmission. This may indicate a slow destination SIEM, a high log volume from the appliance, or a slow connection to the destination SIEM.

Gateway

High

Gateway appliance certificate does not include the profile hostname <n>

The client hostname in the configuration is not present as a SAN in the appliance certificate. Renew the certificate from the Appliances page (System > Appliances).

Gateway

High

cz-sessiond: High watermark event queue

The Gateway is struggling to keep up with firewall rule generation and sessions. Upgrade to version 6.1.x or later, add more Gateways to distribute the load, refactor entitlements to reduce processing demand, or convert dynamic rules to static rules.

Gateway

High

cz-sessiond: No revocation has been received for X secs

Gateway-to-Controller communication is not working, so the Controller cannot push the revocation list to the Gateway and the Gateway cannot pull it from the Controller. Check connectivity between the Gateway and Controller.

Gateway

Medium

The following DNS names are unstable

The DNS names used in entitlements are returning inconsistent results, causing firewall rules to update continuously. This may result in different IPs being resolved on the client versus on the Gateways. Create a DNS policy for the DNS name or domain and add it to the DNS Forwarder configuration, then replace dns://<n> in the entitlement with *.domain.com. This ensures the DNS result sent to the client matches the one used by the Gateway. This change is required for public DNS names.

Gateway

Medium

cz-sessiond: The following applications are reported as unhealthy

The Gateway has flagged one or more applications as unhealthy.

Connector

Low

Connector client X: Waiting for configuration. Connection failed.

The client cannot connect to the Controller. Check the global client profile DNS name and verify that the Connector can resolve it.