System connections

The main functional elements in an AppGate ZTNA Collective are: Controller, LogServer, LogForwarder, Gateway, Connection Broker, Connector, Portal, Metrics Aggregator, and the client.

The control and reporting functions relate to:

• Controller

• LogServer

• LogForwarder

• Metrics Aggregator

The access control and provisioning functions relate to:

• Gateway

• Connection Broker

• Connector

• Portal

• Client

The system is optimized for a distributed network architecture with the different appliance functions deployed separately (or in allowed combinations). To support this distributed architecture, TLS communication is used for all system connections within the Collective and outwards to ZTP. If you plan to make changes to hostnames and IP addresses, refer to Certificates to ensure the TLS connections remain intact.

This section provides information about system connections that exist between the different elements in a Collective allowing other appliances, admins, and clients to connect to appliances.

Admin/API TLS Connection

Controllers and the LogServer. There is an Admin/API TLS connection provided for system management on TCP port 8443. There will be inbound connections established for two reasons:

• By admins wanting to access the admin UI and LogServer.

• By external systems making API calls to the Controller.

Allowed Sources should be used to allow access from only these sources.

All appliances. Include an SSH Server with port 22 access. This can be configured or disabled in System Settings. With the Connector it may not be possible to achieve direct SSH access because of its location. In this case it is still possible to SSH to the appliance but through the tunneled connection from the Connector. Refer to SSH command line administration.

Refer to System security - best practice before configuring Admin/API-to-appliance communications.

Appliance-to-appliance communications

The System TLS Connection is also used for peer-to-peer traffic.

Single packet authorization (SPA) implementation used by AppGate ZTNA for appliance-to-appliance communications can sometimes fall foul of firewalls set to allow all TLS. It is therefore strongly recommended to specifically allow TCP 443 between appliances.

• Connector - There is no incoming traffic allowed from peers. Because there is no incoming external traffic, the Connector can use DHCP and/or sit behind a default gateway/firewall without a resolvable hostname.

• Gateway - There is no incoming traffic allowed from peers. Unlike traditional HA firewalls, Gateways do not communicate with other Gateways.

• Connection Broker - This feature is currently in Beta. For more information, contact your AppGate representative.

• Portal - There is no incoming traffic allowed from peers.

• Controller - Allows incoming traffic from peers. The other members of the Collective will all establish connections to any or all of the Controllers using their Appliance Hostname. A resolvable hostname therefore needs to be configured in System Settings.

• LogServer, LogForwarder, and Metrics Aggregator - Allow incoming traffic from peers. The other members of the Collective will establish connections in order to send audit logs/metrics (for more information, refer to System Logs). A static IP address or resolvable hostname therefore needs to be configured in System Settings.

NOTE

When running HA Controllers, the relevant Controllers need to synchronize. Controllers have specific requirements we recommend you follow.

Client-to-appliance communications

Clients, whether on a user's device, the Connector, or the Portal need to connect to both Controllers and Gateways.

For Controllers

• There is the Profile DNS name which was created when the first Controller was configured. This is shared across all Controllers and passed to the clients in the client profile. It is used to establish connections to Controllers using TLS on port 443 (this can be changed in System TLS Connection if required).

• When using HA Controllers we recommend you use a DNS record with multiple A records. The concept of the Profile DNS name was created with this in mind. All of your Controllers share this DNS name already, so a DNS record for this hostname should be created that contains multiple A records - one for each Controller. The DNS response is the only way that the client knows about all of the available Controllers, thus allowing it to try the next IP address in the list when one is unable to respond. This is an advantage, for instance, when a user can't authenticate to a Controller because the link between that Controller and the IdP is down. The Controller will return a time-out message to the client which then knows to try the next IP address in the list.

For Gateways

• There is a client hostname/IP (set in Secure Tunnel Settings) passed to the clients and is used to establish the secure tunnel to Gateways using TLS on port 443 (this can be changed in System TLS Connection).

  • There is the option to use DTLS for the tunneled traffic (set in the Sites), then client to Gateway communication for that Site will use UDP port 443 (this can be changed in Secure Tunnel Settings). The client hostname/IP will still be used in this case.

  • For HA Gateways, we recommend you use the built in Gateway load balancing capability provided by the AppGate ZTNA system. This allows you to take advantage of the weightings (set in Secure Tunnel Settings). Gateways can be fed by independent networks/ISPs so will provide a complete failover solution without any single point of failure.

Client-to-appliance communications using a Proxy (Proxy Protocol)

The AppGate ZTNA system relies on claims to make access decisions. The use of proxies and load-balancers removes the originator's source IP information from the TCP packet and replaces it with the source IP as the proxy itself. Proxy protocol adds a new header to the TCP packet which includes the originator's source IP. This is used in the AppGate ZTNA system to populate the clientSrcIP claim so 'network' can still be used to make access decisions. AppGate ZTNA supports both version 1 and version 2 of proxy protocol.

Proxy protocol is available in various products including cloud environments such as AWS and GCE.

Proxy protocol support can be enabled in System TLS Connection.

Management network interface

Each appliance can be configured with a management network interface. An appliance needs a minimum of two NICs for this feature to be used. When enabled, by default the following services will be routed to the management interface and will no longer be available via the normal interface:

• SSH Server

• SNMP Server

• Healthcheck Server

• Prometheus Exporter

There is an option to include the following additional service:

• Admin/API TLS Connection

There is also a “free” option which allows any TCP/UDP port to be routed to the management interface if required. This is useful if you are running a customization that exports data from an appliance.  

The management network is configured from the command line using the cz-config command. This configuration includes a "routes" option which will be used instead of any routes specified when the NIC was configured.

System TLS Connection

The System TLS Connection on port 443 is cloaked by single packet authorization (SPA). This means that this port can be exposed to the internet safely and there is no requirement to control access by specific IP addresses or to protect the port behind a firewall.

Even though AppGate ZTNA uses the standard TLS protocol, if the TLS traffic passes through application-aware firewalls (that are configured to allow normal HTTPS://) this might not be sufficient for AppGate ZTNA to operate correctly.

AppGate ZTNA uses two TLS custom extensions; the first comprises the SPA handshake, and the second includes instructions as to how the Gateway should direct the packet. Typically you should allow all TLS which should be sufficient for clients to connect as the extensions are part of the normal 'TLS ClientHello' packet. If allowing TLS does not work, then you should specifically allow TCP 443 to the appliance.

Firewalls should not be attempting to perform any “man in the middle” adjustments to the traffic such as messing with any TLS certificates.

Refer to User/device troubleshooting for more details about how to spot when troublesome firewalls are messing with System TLS Connections such as clients being unable to connect.

NOTE

Because the System TLS Connection is cloaked by SPA, most normal quick checks will no longer work and the port will appear to be unavailable.

ZTP

AppGate offers a range of facilities through the ZTP cloud service. To be able to use AppGate's ZTP service, all the Controllers must be able to reach *.appgate.net.

Appliance firewall

Additionally, all appliances include the usual firewall found in Linux distributions that acts on all System connections. This provides basic stateful firewall functionality and can be used to further control traffic to or from appliances within the AppGate ZTNA Collective. Within the Add/Edit Appliances page, specific filters for <Allow Sources> and/or <Allow Destinations> can be defined in:

• SSH Server

• SNMP Server

• Healthcheck Server

• Prometheus Exporter

• System TLS Connection

• Inbound Peer Connection

• Admin/API TLS Connection

• HTTPS Settings

• Secure Tunnel Settings (allow destinations)

• DNS Forwarder (allow destinations)

Connections table

The table below provides a summary of the required network ports that need to be open for all the functions found within a newly deployed Collective. Alternative configuration settings may require the ports shown in (brackets).

The Interface Schematic in the Appendix illustrates interface connections in a typical installation.