--- title: "Appliance healthchecks and diagnostics" slug: "appliance-healthchecks-and-diagnostics-v7-0" description: "Describes the Site, appliance, and functional healthchecks that appear on the dashboard in the Sites widget and the Appliances widget." tags: ["Appliance Troubleshooting", "Healthcheck Server", "Healthchecks", "Troubleshooting"] updated: 2026-09-21T13:56:05Z published: 2026-09-21T13:56:05Z canonical: "support.appgate.com/appliance-healthchecks-and-diagnostics-v7-0" --- > ## Documentation Index > Fetch the complete documentation index at: https://support.appgate.com/llms.txt > Use this file to discover all available pages before exploring further. # Appliance healthchecks and diagnostics The Collective performs about 50 Site, appliance, and functional healthchecks. Results appear on the dashboard in the Sites widget and the Appliances widget. From either widget, go to **Appliance Health Details** to view warnings or errors. This topic describes the healthcheck messages you may see and the action to take for each one. For SSH commands and other diagnostic tools, see [Appliance commands reference](/v1/docs/appliance-commands-reference). --- ## Errors The following table describes error-level healthcheck messages and the action to take when your Collective reports one. | Source | Urgency | Message | Action | | --- | --- | --- | --- | | Appliance | High | Current Controller cannot reach the appliance on appliance-hostname:443 | Verify the appliance is running. Verify that the Controller hostnames are resolvable on the appliance and that port 443 is open to the Controllers. Verify the time on the appliance. SSH to the appliance and run `nc -zv controller-hostname 443`. This command returns `succeeded` if the Collective is in TCP-SPA mode. If it does not, check the network firewall for man-in-the-middle interference. | | Appliance | High | I/O Stalled | The disk I/O stalled for several seconds, indicating either an availability issue or a capacity problem with the underlying storage system. Verify hardware diagnostics or check your hypervisor stack for storage issues. | | Appliance | Low | I/O Error | The underlying storage system of the hardware or hypervisor is reporting an error. Verify that storage is working correctly, run storage diagnostics, and check for read or write issues on the underlying disk. I/O errors can lead to data corruption. | | Appliance | Medium | GeoIP database missing | The appliance cannot download GeoIP data from https://bin.appgate-sdp.com or https://updates.maxmind.com. Verify that the appliance has internet access and that DNS is working correctly. If GeoIP data is not required, or external server connections are not permitted, disable GeoIP in **Global Settings** (**Settings** > **Global Settings**). | | Appliance | Medium | Failed to read NTP status | NTP status cannot be verified. On the appliance, run `sudo ntpq -np` and check for errors. This issue is most likely caused by a DNS or connectivity problem that prevents the appliance from reaching the configured NTP servers. | | Appliance | High | Failed to perform healthcheck for | The healthcheck service is not running on this appliance. Check `cz-configd` logs for details. | | Appliance | Medium | Not connected to any Controller | The appliance cannot reach any Controllers in the Collective. Verify that the appliance can reach Controller TCP port 443 (default). If UDP-SPA is enabled, verify that the appliance can connect to UDP ports 53 and 443 on the Controllers. Verify that the time is synchronized across all appliances. | | Appliance | High | Customization error | The appliance has a broken customization script. Download the appliance logs and check `logs_by_daemon/cz-customization.log`. | | Appliance | High | Stuck initializing cloud instance | The appliance is expecting cloud-init information but is not receiving it. Verify your network settings and confirm with your cloud provider that it is sending cloud-init data. Verify that DHCP settings for DNS and the default gateway are enabled, as most platforms require these to receive cloud-init information. | | Appliance | Medium | The following services are not running | One or more daemons that are expected to be running have not started. Sign in to the appliance and check the daemon status: `sudo systemctl status <daemon name>`. | | Appliance | Medium | High volume usage [X%] | The specified volume is more than 90 percent full. Check what is consuming disk space and remove unnecessary files, such as old core dumps. | | Controller | High | Unable to connect dbd instance | The Controller cannot reach the database daemon. Check `cz-dbs` for status and contact support. | | Controller | High | Unexpected state for running Controller | Contact support. | | Controller | High | IP Pool has X IPs allocated out of Y (90 percent or higher usage) | The Controller is running low on IPs from the IP pool. Go to **IP Pools** (**Identity** > **IP Pools**) and compare currently used IPs against the total pool size. If used IPs are close to the total size, either reduce the lease time to free up IPs or add additional ranges. When adding ranges, verify that they are routed correctly for each Site not using SNAT. If used IPs are at least 50 percent lower than the total size, users are likely reaching only one Controller; resolve the connectivity issue so the new Controller can assign the unused IPs. | | Gateway | High | Failed to query cz-sessiond for status | The `cz-sessiond` daemon is not working correctly. Restart it with `sudo systemctl restart cz-sessiond`. | | Gateway | High | Very high number of active connections | The connection tracking table has reached 95 percent capacity. At 100 percent, users may experience dropped application sessions. Check conntrack settings with `sudo sysctl -a | | Portal | High | Failed to query cz-nginx@urlaccess for status | Run `sudo systemctl status cz-nginx@urlaccess` to check that the service started correctly. | | Portal | High | cz-nginx@urlaccess: Shared memory size is not enough to save all the HTTP up action objects and the auxiliary data | Add more memory or reduce the number of HTTP up actions. | | LogServer | Medium | OpenSearch is down or starting up | Run `sudo systemctl status cz-opensearch` to check that the service started correctly. | | LogServer/LogForwarder | Medium | cz-logd: Unable to connect to Elasticsearch | LogServer or LogForwarder cannot communicate with Elasticsearch. Check whether Elasticsearch is down. | | LogServer/LogForwarder | Medium | cz-logd: Unable to prepare POST request for inserting data into Elasticsearch | LogServer or LogForwarder cannot post inserts into Elasticsearch. Check connectivity to the HTTP port of Elasticsearch and verify that the Elasticsearch configuration matches the LogForwarder configuration. | | LogServer/LogForwarder | Medium | cz-logd: Unable to create health request for Elasticsearch | LogForwarder or LogServer cannot query health information from Elasticsearch. Check Elasticsearch connectivity and configuration. For LogServer, also verify that the Elasticsearch service is running. | | LogServer/LogForwarder | Medium | cz-logd: Unable to create index into elasticsearch, got status: X | LogServer or LogForwarder cannot create indexes in Elasticsearch. Check the Elasticsearch configuration. | | LogServer/LogForwarder | Medium | cz-logd: Elasticsearch status is not green/yellow | The Elasticsearch status is not green. Check the Elasticsearch server status and restore it to green or yellow. | | LogForwarder | Medium | cz-logd: Unable to get stream, does it exist?, streamname: X, details: Y | LogForwarder is configured with a stream that does not exist in AWS. Verify the configured stream name or create it in AWS. | | LogForwarder | Medium | cz-logd: Unable to get delivery stream, does it exist?, streamname: X, details: Y | LogForwarder cannot get the delivery stream. Check IAM roles for the Kinesis output. | | LogForwarder | Medium | cz-logd: Could not compile filters, X | LogForwarder has a filter configured that is not compiling. Check the filter configuration for the LogForwarder. | | LogForwarder | Medium | cz-logd: No credentials provided | LogForwarder Kinesis output has no credentials or invalid credentials. Check the Kinesis LogForwarder configuration. | | LogForwarder | Medium | cz-logd: Unable to get AWS region, details: X | LogForwarder cannot retrieve information about an AWS region. Verify that the correct region is configured for the Kinesis output. | | LogForwarder | Medium | cz-logd: Unable to create AWS session, details: X | LogForwarder cannot create an AWS session using the AWS SDK. Check AWS credentials and region configuration. | --- ## Warnings The following table describes warning-level healthcheck messages and the action to take when your Collective reports one. | Source | Urgency | Message | Action | | --- | --- | --- | --- | | Appliance | Low | GeoIP database was last updated X days ago | The appliance did not receive the latest GeoIP data from https://bin.appgate-sdp.com or https://updates.maxmind.com. Verify that the appliance has internet access and that DNS is working correctly. To force an update manually, run `sudo /etc/cron.daily/geoIpDbUpdate --force`. | | Appliance | Medium | High volume usage [X%] | The specified volume is more than 75 percent full. Check what is consuming disk space and remove unnecessary files, such as old core dumps. | | Appliance | Medium | Certificate with subject for has expired | An appliance certificate has expired. The appliance stops accepting connections when this occurs. Certificates renew automatically from version 6.1 onward, so this warning appears only if the appliance was offline during renewal. | | Appliance | Medium | Certificate with subject for is expiring | An appliance certificate expires within 30 days. Select the renew certificate option in **Appliances** (**System** > **Appliances**). Renewing the certificate restarts all services on this appliance. | | Appliance | Medium | The following services have debug logs enabled | Running with debug logs enabled may affect performance. Switch back to normal log levels once troubleshooting is complete. | | Appliance | Low | Configuration from Controller is incompatible with this appliance | The configuration from the Controller does not match the configuration format of this appliance. This is likely caused by a version incompatibility. | | Appliance | Low | cz-ffwd: Unable to connect to X@X (X) | The appliance cannot establish a WebSocket connection to the LogServer or LogForwarder. Check connectivity between appliances (default TCP port 443, UDP ports 53 and 443). | | Appliance | Low | This system has a CD drive attached | The appliance is running on VMware and still has a CD drive attached. Remove the attached CD drive from the virtual machine in the VMware console. | | Controller | High | There are more X than your license allows | The number of users exceeds the licensed limit. Access is granted to the first set of licensed users only. Contact support or sales to update your license. | | Controller | High | IP Pool is too small to be utilized by this Controller | The assigned IP pool has fewer IPs than the number of Controllers. For example, a /30 pool with six Controllers produces this error. Assign a larger IP pool. | | Controller | High | Controller appliance certificate does not include the client profile DNS name | The client profile DNS name is not present as a subject alternative name (SAN) in the appliance certificate. Renew the certificate from the **Appliances** page (**System** > **Appliances**). | | Controller | Medium | Optimizer is not active because this Collective is not registered to ZTP. | Register the Collective to ZTP. | | Controller | Medium | Optimizer ZTP service is not available. | Check the status of your ZTP service. | | Controller | Medium | Optimizer is not active because there is no active LogServer or LogForwarder in this Collective. | Add a LogServer or LogForwarder to the Collective. | | Controller | Medium | X of the user licenses are in use | You are approaching the user license limit. Contact support or sales to update your license count. | | Controller | Medium | X of the Portal licenses are in use | You are approaching the Portal license limit. Contact support or sales to update your license count. | | Controller | Medium | X of the service licenses are in use | You are approaching the service license limit. Contact support or sales to update your license count. | | Controller | Medium | Controller is running in maintenance mode | The Controller entered maintenance mode during an upgrade. If the upgrade was cancelled and the node remains in maintenance mode, run `sudo cz-config set -j Controller/maintenance false` to exit maintenance mode, only after confirming the upgrade has been cancelled. | | Controller | Medium | Database node not replicating | This Controller cannot replicate the database with another Controller. Check connectivity between the two Controllers; bidirectional connectivity is required. | | Controller | Medium | IP Pool has X IPs allocated out of Y (75 to 90 percent usage) | The Controller is running low on IPs from the IP pool. Go to **IP Pools** (**Identity** > **IP Pools**) and compare currently used IPs against the total pool size. If used IPs are close to the total size, either reduce the lease time to free up IPs or add additional ranges. When adding ranges, verify that they are routed correctly for each Site not using SNAT. If used IPs are at least 50 percent lower than the total size, users are likely reaching only one Controller; resolve the connectivity issue so the new Controller can assign the unused IPs. | | Controller | Low | BDR conflict | Different Controllers have conflicting versions of data, likely caused by a temporary network connectivity issue. Most conflicts resolve automatically by accepting the latest record update. To resolve remaining conflicts, run `sudo cz-config bdr resolve-conflicts`. If this warning continues to appear, contact support. | | Controller | Low | The following entitlements are using the deprecated Risk Based Access feature | Migrate these entitlements to Condition Based Access, which provides the same functionality using conditions with risk score criteria. | | LogForwarder | Medium | cz-logd: Not connected to X | A LogForwarder output named X is not connected. Check connectivity from the LogForwarder to the log destination. | | LogForwarder | Medium | cz-logd: Unable to perform log-retention | The LogForwarder cannot remove indexes from Elasticsearch. Check Elasticsearch connectivity and configuration. | | LogForwarder | Medium | cz-logd: Failed to send logs to X | Connectivity for an HTTP-based output is not working. Check connectivity from the LogForwarder to output destination X. | | LogForwarder | Medium | cz-logd: Kinesis was unable to handle all records | The Kinesis output is being throttled. Configure additional shards in AWS. | | LogForwarder | Medium | cz-logd: Kinesis error codes: X | AWS is returning an error code for the Kinesis output. These errors are typically IAM-related. Check and correct the AWS IAM configuration. | | LogForwarder | Medium | cz-logd: Firehose was unable to handle all records | The Kinesis Firehose output is being throttled. Configure more resources on the AWS side to handle the load. | | LogForwarder | Medium | cz-logd: unable to connect to LogForwarding destination | The LogForwarder has a TCP-based output configured that cannot connect. Check connectivity to the configured output. | | LogForwarder | Medium | cz-logd: TCP output is slow | The rate of log generation exceeds the rate of log transmission. This may indicate a slow destination SIEM, a high log volume from the appliance, or a slow connection to the destination SIEM. | | Gateway | High | Gateway appliance certificate does not include the profile hostname | The client hostname in the configuration is not present as a SAN in the appliance certificate. Renew the certificate from the **Appliances** page (**System** > **Appliances**). | | Gateway | High | cz-sessiond: High watermark event queue | The Gateway is struggling to keep up with firewall rule generation and sessions. Upgrade to version 6.1.x or later, add more Gateways to distribute the load, refactor entitlements to reduce processing demand, or convert dynamic rules to static rules. | | Gateway | High | cz-sessiond: No revocation has been received for X secs | Gateway-to-Controller communication is not working, so the Controller cannot push the revocation list to the Gateway and the Gateway cannot pull it from the Controller. Check connectivity between the Gateway and Controller. | | Gateway | Medium | The following DNS names are unstable | The DNS names used in entitlements are returning inconsistent results, causing firewall rules to update continuously. This may result in different IPs being resolved on the client versus on the Gateways. Create a DNS policy for the DNS name or domain and add it to the DNS Forwarder configuration, then replace `dns://<n>` in the entitlement with `*.domain.com`. This ensures the DNS result sent to the client matches the one used by the Gateway. This change is required for public DNS names. | | Gateway | Medium | cz-sessiond: The following applications are reported as unhealthy | The Gateway has flagged one or more applications as unhealthy. | | Connector | Low | Connector client X: Waiting for configuration. Connection failed. | The client cannot connect to the Controller. Check the global client profile DNS name and verify that the Connector can resolve it. |