High availability

Prev Next

AppGate ZTNA includes built-in high availability (HA) mechanisms at multiple levels. This section describes those mechanisms and provides guidance on external load balancer limitations and Gateway failover behavior.


External load balancers

You can use external load balancers in front of Controllers and Gateways, but doing so introduces limitations. Two of AppGate ZTNA's core security features—SPA and mutual TLS (mTLS)—restrict the use of several load balancer features, and some advanced HA features are unavailable entirely. AppGate recommends using the internal HA mechanisms for both Controllers and Gateways, which are designed to optimize scalability, security, and performance.

SPA limitations

UDP-TCP SPA mode cannot be used with an external load balancer because the load balancer cannot handle both UDP and TCP traffic. Load balancers also lack an equivalent cloaking feature, which makes them visible on the internet and undermines a core principle of the software-defined perimeter (SDP) model.

TCP SPA mode with mTLS prevents the use of any application-layer load balancer, because together these features protect against man-in-the-middle attacks. The load balancer must operate exclusively at the DNS or TCP/IP layer.

Supported load balancer configurations

Two configurations are supported:

  • Round-robin DNS. Set the Client Hostname/IP (Gateway) or Profile DNS name (Controller) to return multiple IP addresses. The DNS service selects an address based on round-robin or predefined criteria such as geolocation. Configure the load balancer to use the Healthcheck Server to verify appliance availability. In this configuration, the client has no list of available IP addresses. If it receives a timeout from the connected Controller, the user cannot authenticate.

  • TCP/IP layer load balancer. Set the DNS record for Client Hostname/IP (Gateway) or Profile DNS name (Controller) to point to the load balancer. The load balancer forwards traffic to an appliance based on predefined criteria. Configure the load balancer to use the Healthcheck Server to verify appliance availability. As with round-robin DNS, the client has no list of available IP addresses, so a Controller timeout prevents user authentication.

If you use a load balancer with Controllers and configure it to direct all traffic to the primary Controller first, increase IP pool sizing significantly to account for how pools are split across Controllers.


Gateway stateful failover and roaming

AppGate ZTNA does not require stateful firewalling because only authenticated users connect on one side and protected hosts sit on the other. Separate inbound and outbound rules prevent reverse connections from the protected host side.

When a client establishes a secure tunnel with a Gateway, it pushes the user's entitlement and claims tokens to the Gateway. The Gateway starts a firewall service on a separate thread and builds an initial set of firewall rules from the entitlement token.

If the connection is interrupted, the client attempts to reconnect—to the same Gateway if only one serves the Site, or to another Gateway in the Site. After reconnecting, the client pushes the tokens again, and the Gateway rebuilds the firewall rules from the same entitlement token. Established network connections continue without interruption, and the failover is transparent to the user and to applications on the device.

Because there is no Gateway-to-Gateway communication in AppGate ZTNA, the original Gateway retains the user's session for five minutes after the client connects elsewhere, then terminates it.

NOTE

Stateful failover at the TCP/IP level is not possible when SNAT is enabled, because the protected host must see the failed-over connection from the same IP address. If a Site has only one SNAT-enabled Gateway, the client resumes communication seamlessly after a connection interruption or roaming event.


AppGate ZTNA HA

AppGate ZTNA uses linearly scalable, stateless appliances. The client manages all Controller and Gateway HA mechanisms using client profiles (for Controllers) and entitlement tokens (one per Site).

The client uses a built-in round-robin load-balancing algorithm to select Controllers and Gateways. If an appliance is busy or suspended, it rejects the connection and the client tries again. Gateway disconnections trigger an automatic reconnection with minimal interruption—users do not need to sign in again.

AppGate ZTNA also supports advanced mechanisms including cross-Collective HA, which allows clients to fail over to a different Collective when no Controllers are available. HA functionality extends to Connectors (see Adding a Connector) and multi-LogForwarder HA (see Audit logs).


Controller and Gateway HA mechanisms

The following internal HA mechanisms go beyond what external load balancers can provide. Read this section together with the Using HA Controllers section.

Internal HA operates in the following situations:

  • The client cannot establish an initial connection to a Controller or Gateway. This can result from UDP-TCP SPA behavior. Configuring a SPA override on an alternative appliance may resolve this.

  • The Controller instructs the client to fail over to a different Controller due to high load, exhausted IP pool space, or LDAP timeouts. This requires multiple Controllers per Collective.

  • Cross-Collective HA, where the client tries a different Collective when no Controllers are available. This requires a profile group and a device policy that includes client profile settings.

  • A Gateway rejects a new client connection. The client fails over to another Gateway in the load-balance group.

  • A Gateway becomes unresponsive. The client fails over to another Gateway in the load-balance group. This requires multiple Gateways per Site.

  • All Gateways in one load-balance group become unresponsive. The client fails over to Gateways in a different load-balance group within the same Site. This requires special Gateway weightings.

  • A Gateway reports that a monitored action is unhealthy. The client fails over to a healthier Gateway. This requires action monitoring to be configured.

  • No Gateways are available on a Site after a minimum of 30 seconds. The client switches to the fallback Site. This requires configuration in both the Site and the policy.

Controller failover using internal load balancing

The internal load-balancing mechanism relies on the profile DNS name created when you create the first Controller. This name is automatically shared across all Controllers in the Collective. Create a multiple A-record DNS entry for this name—one record per Controller. The name appears in Global Settings as the global DNS name, which you can edit if needed.

The AppGate ZTNA client reads the profile DNS name from the client profile. A correctly configured DNS query returns multiple IP addresses, which the client tries one by one until one responds.

NOTE

If you use a new DNS name (global or custom) in a new client profile, you must renew each Controller's certificate manually before the profile can be used.

Cross-Collective HA using profile groups

A profile group contains multiple client profiles from any Collective. Each profile is assigned a weighting that works similarly to Gateway weightings. When no Controllers for a given client profile are available, the client checks for another profile in the same group. If one exists, the client selects it based on its weighting and tries its Controllers.

Profiles with equal weightings distribute users across Collectives evenly.

Cross-Collective HA can also provide an alternative authentication path to a single Collective. For example, assign a higher weighting to an OIDC client profile and a lower weighting to an LDAP profile. If OIDC becomes unavailable, the Controllers appear unreachable and the client tries the LDAP profile, where users can sign in.

Gateway auto-suspend

When a client tries to connect to a Gateway, the Gateway may reject the connection if it has been manually suspended or automatically suspended due to heavy load. The following thresholds trigger automatic suspension and resumption:

Metric

Description

Suspends at

Resumes at

eventQueueUsage

Percentage of the event queue used (events awaiting processing by sessiond)

70%

50%

javaHeapUsage

Percentage of sessiond memory used versus maxiumum heap

90%

75%

numberOfSessions

Number of user sessions on the Gateway

8000

7500

systemMemoryUsage

Percentage of total memory used on the Gateway

85%

70%

These are the built-in default values. Adjust them using cz-config if needed.

Gateway failover redundancy

To ensure all clients can connect if one Gateway goes offline, keep Gateway utilization during normal operation below the levels shown in the following table. Staying within these levels prevents auto-suspend:

Number of Gateways

Event queue usage

Heap usage

Number of sessions

Memory usage

2

35%

45%

4000

43%

3

47%

60%

5350

57%

4

53%

68%

6000

64%

5

56%

72%

6400

68%

Adding a third or fourth Gateway significantly improves the acceptable utilization during normal operation. Calculate the acceptable utilization using the following formula: (x × (n − 1)) / n, where x is the suspend threshold (for example, 90%) and n is the number of Gateways during normal operation.

Gateway failover within the load-balance group

When the client connects to a Site, it selects a Gateway based on the weighting factor, which biases the random selection process. Once connected, the client monitors traffic and sends a keep-alive message every five seconds when no application traffic is detected. When the client detects a connection interruption, it attempts to connect to an alternative Gateway for that Site.

Weighting

Each Gateway is assigned a three-digit Client Tunneling - Load Balance Weighting Factor. Gateways with equal weights receive equal traffic. The ratio between weights determines the distribution—not the absolute values. For example, weights of 100 and 50 produce a 2:1 distribution.

Example

Three Gateways have weights of 10, 10, and 5:

Gateway

Weight

Probability of being tried first

A

10

10/25*100 = 40%

B

10

10/25*100 = 40%

C

5

5/25*100 = 20%

If Gateway B is offline:

Gateway

Weight

Availability

Probability of being tried first

A

10

100%

10/15*100 = 66.6%

B

10

0%

0%

C

5

100%

5/15*100 = 33.3%

Gateways A and C must handle the additional load while B is offline.

Gateway failover to another load-balance group

Assigning a weight of zero to a Gateway places it in reserve. The client always tries higher-weighted Gateways first. It tries zero-weighted Gateways only when no others are available. Use this to configure reserve appliances.

NOTE

A zero-weighted Gateway will typically have some clients connected at any given time, due to temporary connection issues with higher-weighted Gateways or because one or more Gateways were offline during an upgrade..

Example

Gateways A and B use the primary ISP. Gateways C and D use the reserve ISP:

Gateway

Weight

Probability of being tried first

A

100

100/200*100 = 50%

B

100

100/200*100 = 50%

C

0

0/200*100 = 0%

D

0

0/200*100 = 0%

Priority Groups

If a Site has many Gateways, working through all higher-weighted Gateways before trying reserve Gateways can cause a lengthy outage. Priority groups reduce this by triggering a move to the next group after just two sequential failures within the current group.

Priority groups activate automatically when every Gateway on a Site uses a weighting factor of four or more digits (greater than 999). The last three digits represent the weighting factor. The preceding digits represent the priority group number. Higher group numbers are tried first.

Example

Gateway

Weight

Group

Notes

A

10100

10

33% probability within group

B

10100

10

33% probability within group

C

10100

10

33% probability within group

D

1100

1

Used only after two failures in group 10

E

1100

1

Used only after two failures in group 10

F

1100

1

Used only after two failures in group 10

NOTE

If Gateways on a Site have weights of 500, 2000, and 2500, priority groups do not apply. The system treats these as standard weighting factors, resulting in a 10%, 40%, and 50% distribution.

Gateway failover triggered by a monitored action

When configuring an entitlement, you can enable monitoring for HTTP and TCP Up action types. The system checks monitored applications for return traffic. If no return traffic arrives within the response timeout period, the system marks the application as unhealthy. Monitoring a commonly used application can also detect general network failures—for example, when a network switch handling a Gateway's inside connection fails.

When the client receives an unhealthy application report from a Gateway, it checks its unhealthy report history from recent connections to other Gateways. The client retains unhealthy reports for 10 minutes, after which it assumes the Gateway is healthy again. The client then fails over to the healthiest available Gateway. If two Gateways are equally healthy, the client uses the normal weighting process. A zero-weighted Gateway is used if it is the healthiest option the client knows about.

Site failover to the fallback Site

The client attempts to find a working Gateway on a Site for a minimum of 30 seconds. After this period, the client switches to the fallback Site. Because the client controls this process, it does not matter if the affected Site also hosts Controllers.

NOTE

The fallback Site feature does not support chaining Sites. You can have only one primary and one fallback Site for a given entitlement.

This mechanism suits organizations that operate a WAN across multiple locations with external access through Sites at different locations. The fallback Site can serve as an alternative for all users of a given Site, or as a back-channel for administrators to access the WAN and resolve an outage. Enable Use fallback Site in the relevant policies to control which users can use the fallback Site. This policy-based model allows you to size the fallback Site appropriately for RAM and CPU.

RAM sizing is particularly important. During normal operation, sessiond configures all entitlements for users of the Site, but vpnd only applies those entitlements when users connect. This means the fallback Site uses approximately 50 percent of the additional RAM required for the extra entitlements during normal operation, rising to 100 percent when failover occurs. Insufficient RAM and CPU cause the fallback Site to become overloaded when it absorbs users from the failed Site. Also ensure that the required routes are in place on the fallback Site. For sizing guidance, see Instance sizing.