Auto-scaling Gateways

All cloud providers have automated means of scaling the number of instances of a specific appliance type up and down. The use of virtual appliances means it is possible to auto-scale the number of Gateways within an AppGate ZTNA Site based on demand. Auto-scaling can be triggered by metrics such as memory and CPU which are normally monitored by the cloud provider.

User Data (AWS), Custom Data (Azure) or Startup Scripts (Google Cloud) are used to create an appliance configuration for the newly started instances based on a template which is also used to seed itself. To take full advantage of this, the Controller needs a way to remove unwanted Gateways form a Site as well as just adding them. To achieve this there needs to be two process integrations:

• Up-scale: Create a new appliance entry and join the AppGate ZTNA Site by seeding that appliance when the auto-scaling group spawns a new instance.

• Down-scale: Delete the appliance entry when an instance is removed from the auto-scaling group.

AWS

AWS provides a feature called auto-scaling groups to automatically increase or decrease the amount of instances running based on some metrics such as CPU or memory load. See:

https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html

https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scale-based-on-demand.html

GCP

GCP provides a feature called instance groups to automatically increase or decrease the amount of instances running based on some metrics such as CPU or memory load. See:

https://cloud.google.com/compute/docs/tutorials/high-scalability-autoscaling

Azure

Azure provides a feature called virtual machine scale sets to automatically increase or decrease the amount of instances running based on some metrics such as CPU or memory load. See:

https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/autoscale-common-metrics#guest-os-metrics-linux-vms

Before you start

To fully automate auto-scaling, a startup script is passed to the AppGate ZTNA instance using the cloud provider's API. This script creates an appliance configuration for the newly started instance based on a template which is also used to seed itself. Since you can only pass a single base64 blob (the script), it needs to contain all the necessary parameters to create a Gateway and the seed configuration. In AWS and Google Cloud, this concept is often referred to as user data. Microsoft Azure has a similar feature called custom data.

To allow this to work, a number of things have to be configured in AppGate ZTNA before you can start. In addition to knowing the Controller's hostname or IP and the admin port 8443. If you have a shared hostname (DNS round robin) configured with multiple Controllers then use this.

Configure the auto-scale Site

Create an auto-scale Site as auto-scaling is configured on a per-Site basis. Assign a tag such as: siteX-for-auto to the Site, which can later be used restrict admin access to this auto-scale Site. To use the script you will need to get the the Site id. This is visible in the URL when navigating to the Site in the admin UI. For example, given the Site URL https://mycontroller.com:8443/ui/sites/edit/750f210a-1c42-4d27-b568-4a8767ef2790, the site id is 750f210a-1c42-4d27-b568-4a8767ef2790.

Configure the auto-scale appliance

Create a new Gateway appliance assigned to the auto-scale Site.  This will become a base template for the auto-scaling process which will then vary only parameters such as hostname. Configure it with all the usual settings except a use bogus but unique hostnames such as: appliance-name.example.com for the appliance.  Even if you are using IP addresses in favor of hostnames just use a bogus hostname here. Assign a tag such as: siteX-auto to the appliance, which can later be used restrict admin access to only the templated and auto-scaled appliances.  

NOTE

Do not activate the appliance. This configuration will be used as a template and it is the new instances created by the template that will end up being activated.

Configure the auto-scale admin user

Create an admin user called autoscaling; you can use the Local user database for this or an external identity provider. Set a strong password, preferably randomly generated. Exempt that user from MFA using the MFA for Admin page. To use the script you will need username and password.

Configure the auto-scale admin role

The auto-scaling script needs various rights, for instance to view the template appliance for the given Site, create a new appliance and export it's seed configuration. Create an admin role called siteX-autoscaling and add the following privileges:

• View all appliances tagged with siteX-auto & Sites tagged with siteX-for-auto  

• Create all appliances with tag siteX-auto

• Export all appliances with tag siteX-auto

• Delete all appliances with tag siteX-auto (for down-scaling)

• Edit appliances with tag siteX-autoscaling-instance (If using deactivation during up or down scaling)

• Assign Function Gateway on all appliances.

Configure the auto-scale Policy

Create a new Policy for that user and assign it the siteX-autoscaling role. For the policy assignment add the following two criteria:

• Identity provider is the identity provider used for for the autoscaling user

• Username is autoscaling

Get CA certificate

If you are using the admin interface (port 8443) with a public cert then you do not need to get the Controller's CA certificate.

Implementing auto-scaling

Using the cloud template and startup script

User Data (AWS), Custom Data (Azure), or Startup Scripts (Google Cloud) are used to create an appliance configuration for the newly-started instance based on a template which is also used to seed itself. AppGate ZTNA uses the following mechanisms on the different cloud platforms:

• AWS: Using instance userdata. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig.html

• GCP: Using virtual machine startup script. See https://cloud.google.com/compute/docs/startupscript

• Azure: CustomData. See https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview

NOTE

Startup scripts are made available to the VM only during first boot/initial setup (provisioning). Provisioning is the process where VM Create parameters (for example, hostname, username, password, certificates, custom data, and keys) are made available to the VM.

See the examples below to help you create the required scripts for your own environment.

Upscaling

If you set up your instance group correctly with your startup script, it will be executed when the new appliance starts. The instance will find the appliance template for its Site as defined on the Controller and create a new appliance configuration based on it. It will then export the configuration, seed itself, and perform the upscaling. The template is selected among the Gateways in the given Site. Inactive appliances tagged with template are chosen in priority as the appliance template.

Downscaling

When the instance is shutdown it will either deactivate or delete the entry from the Controller.

Sharing the client hostname

By default the script will create an appliance configuration using the appliance hostname. If you want it to use the Client Hostname/IP from the template Gateway configuration you can pass the --share-client-hostname. This is especially useful if you have configured your autoscaling group to use a load balancer.

Hostname Conflicts

If a Gateway with the same hostname already exists in the Site, it will be deactivated and the entry reused for this instance. If the Gateway is already seen as activated by the Controller, the autoscale script will exit and do nothing.

The script

The Gateway Autoscale Script is hosted on all appliances and run from a cloud instance's startup script. There are various options that you can specify when configuring autoscaling:

appgate-autoscale.py COMMAND [OPTION1] [OPTION2] [OPTION...] CONTROLLER

CONTROLLER - should be the primary Controller's admin interface hostname (or I address).

NOTE

The default port remains 444 even though the Controller uses 8443 for Admin/API access.

Upgrading Autoscaled Gateways

Active autoscaled Gateways should not be upgraded. When a new AppGate ZTNA version is out, a new image will be available for your cloud platform. Point the autoscaling group to this new image and then, one-by-one, terminate the active Gateway instances still running the old version. They will be replaced automatically by Gateway instances running the new version.

If you are using sdpctl to upgrade your other appliances you can use the exclude flag to skip the autoscaling sites. For example:

sdpctl.exe appliance upgrade prepare --image=<upgrade-image-zip-or-url> --exclude site=string

AWS

On AWS this is done by creating a new launch configuration pointing to the new AMI and updating the autoscaling group to point to the new launch configuration.

Advanced setups

You will need to create the startup script for your site manually. Here are some examples that can be used as templates for your own setup.

Setup using a valid Controller certificate and an external credentials manager

This example uses a fake credentials manager similar to HashiCorp's Vault. This concept works for other providers like AWS Security Token Service (STS). The Controller is configured using with the admin interface enabled on port 8443 and a valid certificate.

bash
#!/bin/sh

# Create a script that outputs the password for the auto-scaling admin user.
# The password is fetched dynamically every time it is needed to not expose credentials unnecessarily.
cat >/tmp/password-executable <<EOL
#!/usr/bin/python3
import JSON
import requests
r = requests.get('http://my-credential-manager.example.com/my-autoscaling-secret', params={'token': 'ASDL...ASZSD'})
password = r.JSON()['data']['password']
print(JSON.dumps({"password": password}))
EOL
chmod +x /tmp/password-executable

# Seed this appliance.
# We can skip the --cacert parameter since we are using a valid public certificate on the admin interface.
python3 /usr/share/admin-scripts/appgate-autoscale.py upscale controller.example.com --port 8443 --username siteAWS-autoscaling --password-path /tmp/password-executable --site af4fedf3-5cc2-43fb-aff7-61b67369a505 --file /home/cz/seed.JSON

# Setup appliance to delete itself on shutdown
cat >/var/cache/cz-scripts/shutdown-script<<EOL
#!/bin/sh
/usr/share/admin-scripts/appgate-autoscale.py downscale controller.example.com --port 8443 --username siteAWS-autoscaling --password-path /tmp/password-executable
EOL

Setup with self-signed Controller certificate using admin interface (port 8443)

This example uses a standard setup with the controller using the admin interface on port 8443 using its self-signed certificate. The password is hardcoded in the startup script, which is not recommended in practice.

bash
#!/bin/sh

# Add your Controller's CA certificate.
# This is not necessary if you are using a valid public certificate on the admin interface.
cat >/tmp/cacert.pem <<EOF
-----BEGIN CERTIFICATE-----
MIIFPDCCAySgAwIBAgIJAKJah5YABq/ZMA0GCSqGSIb3DQEBDQUAMBkxFzAVBgNV
BAMTDkFwcEdhdGUgWERQIENBMB4XDTE3MDIyNDE1MTExOVoXDTI3MDIyMjE1MTEx
[...]
zVAmFQL0rHwjonLsbLGBG54idCNTctN6HisBSdRmd7UlVdrABrbiBj42vQ7H8mwU
5qvvc4QH0X+dwU+QWVW0w+MDQNc+9xkTGTKaEiAiQ8WqigT88xUgTVEbSv25f9fm
-----END CERTIFICATE-----
EOF

# Create a script that outputs the password for the auto-scaling admin user
# In production, this should be getting the password from the appropriate credentials manager, for example STS on AWS.
cat >/tmp/password-executable <<EOL
#!/bin/sh
echo '{"password": "password_for_the_autoscaling_admin_user"}'
EOL
chmod +x /tmp/password-executable

# Seed this appliance
# Use the appropriate port, for example if you are using the admin interface on port 10001 pass --port 10001
# Skip the --cacert parameter if using a valid public certificate on the admin interface
python3 /usr/share/admin-scripts/appgate-autoscale.py upscale controller.example.com --port 8443 --cacert /tmp/cacert.pem --username siteAWS-autoscaling --password-path /tmp/password-executable --site af4fedf3-5cc2-43fb-aff7-61b67369a505 --file /home/cz/seed.JSON

# Setup appliance to delete itself on shutdown
cat >/var/cache/cz-scripts/shutdown-script<<EOL
#!/bin/sh
/usr/share/admin-scripts/appgate-autoscale.py downscale controller.example.com --port 8443 --cacert /tmp/cacert.pem --username siteAWS-autoscaling --password-path /tmp/password-executable
EOL