Upgrading appliances

This section describes the process for upgrading the software version of your AppGate ZTNA appliances and reverting to the previous version if required. AppGate ZTNA appliance upgrades are allowed from a maximum of two versions back. For example, you can upgrade to version 6.2 from 6.0 or 6.1. If you are upgrading from 6.1 to 6.5, then you will need to upgrade to 6.3 and from there upgrade to 6.5.

When a new AppGate ZTNA release is available, use sdpctl to perform the upgrade. There will be a check for the minimum supported upgrade version, and for to identify any compatibility issues. Make sure to backup your Controller before allowing the upgrade to proceed.

Before you start

• Read the release notes for specific instructions relating to the version you are upgrading to, including any skipped versions. If any feature deprecations affect your upgrade, address these issues first.

• Note your primary Controller and its hostname. You will connect to its admin interface and will need to reference this Controller if you must revert your upgrade.

• Backup your Controller before allowing the upgrade to proceed and move the backup file off your device. See Backup and restore for additional information.

NOTE

If you created any custom folders, ensure that the folders and any information residing in them is backed up off the appliance.

• Get the appliance upgrade file from the AppGate Support page. The file will be a zip file, for example, Appgate-SDP-6.1.1.img.zip. Do not decompress the zip file before starting. If your operating system automatically un-compresses the file, then try using CURL. Open a terminal window (that supports CURL) and enter the following:

  curl -u username:password https://bin.appgate-sdp.com/x.y/appliance/Appgate-SDP-x.y.z.img.zip -o appgate-sdp-x.y.z.img.zip -k

  The zip file contains a signature.asc file. This pgp signature is used on the appliance being upgraded to perform a verification of the upgrade file before it is applied. If the verification fails, then sdpctl aborts and the administrator is advised of the failure.

  • Place the file in one of the following areas:

    • On your local PC/the machine running sdpctl. It will then be uploaded by the Controller.

    • On a web server the Controller can access. You can download the file directly to the web server from the AppGate Support page

• Ensure that you have access to any VMs or cloud instances, including iRMC access in the case of a hardware appliance.

• Check for any customizations installed on the appliances. Always stop customizations before an upgrade and restart them afterwards.

NOTE

Temporary customizations provided to address an issue that is now fixed in the version you are upgrading to are not tested against the new version and may therefore be incompatible. If you have a temporary customization, remove it and reboot the system before starting the upgrade.

Upgrading a Collective using sdpctl

sdpctl performs an orchestrated upgrade of all the appliances in your Collective. sdpctl can be used to upgrade an entire Collective or a partially-upgraded Collective, with only the required appliances upgraded.

sdpctl has two phases:

• Prepare. Performs distribution and verification of the image to avoid using damaged files.

• Complete. Completes the upgrade.

NOTE

sdpctl will automatically determine from the current version and target version if this is a patch upgrade (such as 6.1.1 to 6.1.2).

The different appliance functions are upgraded in a pre-ordained order; Gateways, Connectors, LogForwarders, and Metrics Aggregators servicing the same Site will not be upgraded at the same time.

NOTE

It is not recommended to perform simultaneous upgrades of appliances using the host on Controller in a large Collective. You are effectively engineering your own denial of service attack on the Controller by instructing 10s of appliances to request a download of a large (<1GB) file. sdpctl includes the --throttle option to prevent overloading the download server with requests, so please use this wisely to prevent sdpctl timing out.

HA Controllers

For patch upgrades, Controllers remain fully active during the upgrade process to prevent causing issues for HA Controllers. The Controllers are upgraded one by one with the primary first, so only one appliance will be out of service for that short time.

When doing a major upgrade, such as 6.1 to 6.3, a different process is used. The primary Controller will put all other Controllers into maintenance mode. In this mode, Controllers continue to service client requests as normal, however no admin or API requests will be accepted. Once the primary Controller has completed its upgrade, it will be able to handle both Client and admin or API requests, and work starts on upgrading the subsequent Controllers.

Quiet periods should be used for upgrades because the Collective needs to complete any outstanding database sync. Once the sync is complete, the remaining Controllers will be upgraded. If any problems are detected, the primary Controller can perform an automated roll-back.

During this period the Collective needs to perform a database migration. While this is occurring, none of the Controllers can accept client connections (this should take less than a minute). From this point, there is no automated capability to roll back in the event of the upgrade failing. As each Controller completes its upgrade, it is taken out of maintenance mode and is available to handle both client and admin/API requests.

If any of the subsequent Controller upgrades fail or time out, then you need to decide if you are going to go forward or back. AppGate recommends going forward at this point, and there is a section on BDR maintenance for attempting to recover from this situation. There are also some more advanced scenarios detailed, however in untrained hands these procedures can compound the problem making it harder to move forward so we strongly recommend you call AppGate Support.

You can also manually perform an HA Controller rollback.

• If you have a single Controller, follow the Reverting Appliances instructions below.

• If you are using HA Controllers and have gone down to a single Controller before starting the upgrade process, follow the Reverting Appliances instructions below, then SSH to the Controller and enter the following command: cz-config bdr force-single-controller-ready

• If you are using HA Controllers and have multiple upgraded HA Controllers, then you must restore from backup. Use the snapshots and/or backups you took before commencing the HA Controller upgrades. If the system configuration has not changed, then you should be able to restore a failed upgrade using these.

To manually roll back an HA Controller outside of the above scenarios, perform the following steps:

1. On all upgraded machines, run the command sudo cz-config switch-partition
   All appliances will now be running the earlier version. However, you still need to restore Controllers from backup due to the database migrations.

   WARNING

   Using switch-partition and then rebooting should never be done to roll back HA Controllers.

2. On all Controllers except the primary Controller, run the command sudo cz-config bdr force-appliance-ready This removes the additional Controllers.

3. On the primary Controller, run the command sudo cz-config restore /path/to/backup/file

   NOTE

   If the Collective has a LogServer on a different appliance, it also needs to be restored from its backup.

4. Now enable the Controllers one by one in the admin UI, waiting for each to become healthy.

To manually reenable Controllers if the upgrade does not complete normally:

• If the primary Controller upgrade fails, then any additional Controllers may be left in maintenance mode. If the cz-configd daemon is accepting events (normal state), SSH to the Controller to disable maintenance mode by entering:

  sudo cz-config set -j controller/maintenance false

• An appliance that appears to remain offline during or after the upgrade process can be brought back online by restarting the cz-config daemon:

  sudo service cz-configd restart

Reverting appliances

NOTE

LogServers with OpenSearch cannot be reverted.

To revert an appliance to the previous software version:

1. Confirm which version of the appliance is running. The system may have automatically defaulted to running the previous version. To confirm which version is running, SSH to the appliance and enter:  cat /usr/share/cz-image/version

   If the upgrade version is running and you wish to revert to the previous version, continue to the next step.

2. Revert to the previous version using one of the following options:

NOTE

If you are reverting a Controller see HA Controllers above.

Option 1 (temporarily change from the default version):

• Reboot the appliance. The GRUB boot menu will list the new and previous versions of the appliance. The new version will be set as the default. Select the other version from the boot menu if required.

NOTE

This will not be a permanent change. The next time you reboot, the default version of the appliance will be started.

Option 2 (change default version):

• To check which partition is currently running, enter:  cat /mnt/state/volume-number This will return the partition number currently running.

• To change the default partition to the other partition, enter: cz-config switch-partition This changes the boot menu to make the original partition the default version

NOTE

Any new data that you have added to the new version of the appliance will not be migrated to the previous version when you revert.

Reverting an entire Collective

To revert an entire Collective, you should perform the steps in the opposite order from the upgrade. At each step, ensure that each appliance reboots and returns to a healthy state using the previous version:

1. Revert each Portal.

2. Revert each Connector.

3. Revert each Gateway.

4. Revert the LogForwarders/Metrics Aggregator.

5. The final step are the Controllers. See HA Controllers above for more information. If you have more than one Controller, wait for each one to return to a healthy state before moving on to the next.

   • If you need to add the Controller function back to any ex-Controllers, make sure all the other Controllers are in a healthy state before proceeding.

6. Using the dashboard, confirm the entire system is at the previous version and is functioning normally

Confirming the version number

After the upgrade or revert has completed, you can confirm the version number of the appliance either by signing in to the terminal window again or from the dashboard in the admin UI.

Terminal window confirmation:

Authorized uses only. All activity may be monitored and reported.

Last login: Tue Mar 30 14:23:44 2021

Welcome to Appgate SDP (GNU/Linux 5.4.0-64-generic x86_64)

appgate 5.3.2-23587-release (image2)

Hint: run 'sudo cz-setup' for appliance management.

Admin UI: