Controller maintenance and troubleshooting

Prev Next

Before adding or removing Controllers, verify that all Controllers have a status of Healthy.

  • If a Controller shows a Busy status, wait for it to recover before proceeding.

  • If a Controller shows a Warning or Error status, resolve the issue before proceeding.

The Appliances page displaying appliance statuses, highlighting one as warning and another as healthy.


Maintenance scenarios

Controllers can become unhealthy due to disk or network issues, particularly when adding or upgrading Controllers. For guidance on those procedures, see Upgrading HA Controllers.

If Controllers are experiencing issues, force‑disable the affected Controller using the following sdpctl command:

sdpctl appliance force-disable-controller [hostname|ID...] [flags]

This command disables the Controller function on the target appliance and notifies the remaining Controllers. This is the recommended first step in most situations.

If issues persist after force-disabling, refer to the scenarios below. These examples assume three high-availability (HA) Controllers: A, B, and C.

Controller fails to join

Symptoms

Controllers A and B are already active. When you attempt to add Controller C from Controller A's admin UI, one of the "add Controller" checks fails.

Remediation

  1. Resolve any connectivity or database insert conflicts.

  2. Enable the Controller function for Controller C and try again.

If Controller C fails to join and the other Controllers remain in a Busy state, the following errors appear:

  • Controller database controller: BDR node missing: C

  • Controller database controller: New IP allocations are disabled

To resolve this:

  1. Disable the Controller function for Controller C from Controller A's admin UI.

  2. On Controller A, run: sudo cz-config bdr update-bdr-group

  3. Review logs from all Controllers, including Controller C, to identify the reason for the failure.

  4. Reenable the Controller function and try again.

Controller fails to leave, but other Controllers think it has left

Symptoms

This can occur due to network latency. Controllers A and B are Healthy, but Controller C reports an error indicating it is still a Controller when it is not.

Remediation

Follow the steps for Removing a Controller while it is inaccessible.

Controller is gone permanently

Symptoms

This can occur if a Controller virtual machine (VM) is deleted. Controllers A and B time out while waiting for Controller C, and the dashboard shows the following error: BDR node not replicating: C

Remediation

  1. Disable the Controller function for Controller C.

  2. On Controller A, run:

sudo cz-config bdr remove-node-record C
sudo cz-config bdr update-bdr-group
  1. Deactivate or delete the appliance record for Controller C.

  2. Create and register a new appliance.

  3. Enable the Controller function on the new appliance and allow time for synchronization.

Removing a Controller while it is inaccessible

Symptoms

When you attempt to remove Controller C from Controller A's admin UI, Controller C is not accessible. After six minutes, the barrier times out and the following errors appear:

  • Controller database controller: Unexpected BDR node: C

  • Controller database controller: New IP allocations are disabled

Remediation

  1. On Controller C, run: sudo cz-config bdr force-appliance-ready

  2. On Controller A, run: sudo cz-config bdr remove-node-record C

  3. On Controllers A and B, run: udo cz-config bdr repartition-ip-allocations

Warning: “New IP allocations are disabled”

Symptoms

Controller C displays the warning “New IP allocations are disabled.”

Remediation

On Controllers A, B, and C, run: sudo cz-config bdr repartition-ip-allocations

Controller remains in maintenance mode after upgrade

Symptoms

Controller C displays “Controller is running in maintenance mode.”

Remediation

On Controller C, run the following command to remove the Controller from maintenance mode manually:

cz-config controller disable-maintenance

Warning: “BDR conflict detected”

Symptoms

BDR conflicts are more likely when you use the admin UI on multiple Controllers simultaneously.

Error message indicating a BDR conflict in the controller database authorization.

Remediation

  1. Use only one Controller for administration.

  2. If entitlement conflicts exist, resolve them by removing entries in the entitlement_condition and action tables that reference the affected entitlement_id.

  3. On affected Controllers, run the following command and follow the instructions it provides: sudo cz-config bdr resolve-conflicts

For the full set of cz-config bdr commands, see Appliance commands reference.