Datadog Dashboard Configuration

The AppGate ZTNA system exports a variety of metrics that can be useful for monitoring real-time events and to facilitate long-term planning. Datadog provides monitors and dashboards that visualize the metrics provided by AppGate. This allows customers of both Datadog and AppGate to integrate AppGate monitoring into existing monitoring infrastructure.

This guide provides a baseline example that can be customized to fit each customer's unique requirements. This example uses the Metrics Aggregator function to provide a single endpoint to pull metrics using the Prometheus protocol. See documentation provided on Datadog's website or the AppGate Admin Guide for variations on these options and further information.

Prerequisites

• An AppGate appliance capable of functioning as a Metrics Aggregator.

  • In this example, this appliance is referred to as appgate-0-ma

  • You can repeat this guide for multiple site-specific aggregators to prevent unnecessary cross-site traffic.

• A system with the Datadog agent installed.

  • In this example, this system is referred to as datadog-agent-0

  • A VM running Ubuntu Linux or similar. The example config below will assume this, but it isn't a requirement.

  • The system must be able to reach the Metrics Aggregator Prometheus Exporter service.

  • Please consult Datadog's documentation to set up the agent.

Download CA Certificate

If you have already completed the process of establishing the trust of the appliance with TLS certificates, then you can skip this step.

HTTPS should be enabled to ensure that agent-to-appliance communication is encrypted using TLS. If you are using the default Controller self-signed root CA, then you need to download that cert file so that the agent can verify the appliance cert.

To download the file:

1. Navigate to System > Certificate Authority.

2. Click the Actions button and select Download Certificate.

See the Certificate Authority documentation for more information.

Configure the Metrics Aggregator Prometheus Exporter

To configure your Metrics Aggregator:

1. Navigate to System > Appliances.

2. Select your Metrics Aggregator appliance.

3. In the Edit Appliance screen, go to the Functions tab and locate the Prometheus Exporter Settings.

4. Under Allow Access, ensure that you are allowing access from the datadog-agent-0 IP address.

5. Select the Enable Basic Authentication checkbox, add a username, and generate a password.

NOTE

This will generate a warning state on the appliance until metrics are pulled from the agent.

For further information on how to configure the Metrics Aggregator function, please see the documentation.

Configure the Datadog Agent

In this example, we will be using the "openmetrics" check that is provided with the Datadog agent. Use the following example config for /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml:

instances:
  - openmetrics_endpoint: https://appgate-0-ma:5556/metrics
    tls_ca_cert: /etc/datadog-agent/appgate-0-ca.pem
    auth_type: basic
    username: datadog-agent-0
    password: <password goes here>
    max_returned_metrics: 50000
    hostname_label: appliance_name
    namespace: appgate
    tags:
      - source:appgate-0-ma
    metrics:
      - ".*"

NOTE

If you already have some OpenMetrics instances configured, you will need to combine this config with your existing ones.

• hostname_label. This will use the value from the Name field configured under Edit Appliance > System Settings as the "host" name under Datadog.

• tls_ca_cert. A path where you need to copy the root CA that was exported previously, if needed.

• max_returned_metrics. Suggested value: 50000. The default value of 2000 may work for some configurations, but a larger value may be needed if there are many appliances or a configuration that would generate a lot of tag values.

• username and password. Values that were added to Allowed Users in the Prometheus exporter settings.

Please see the Datadog documentation for further details here and an example config.

Optional: Add tags to the hosts in Datadog

The OpenMetrics integration will automatically add tags to the metrics it receives. However, for "Host" objects to be queried based on common AppGate tags in the Datadog UI, they must be tagged separately. For example, this will enable filtering by Collective ID in the Host Map, found in the infrastructure section of Datadog or in the Overview section of the AppGate-provided Dashboard.

For a small number of appliances, tags can be manually set through the UI. However, if you have a large number of appliances, using a script to call the API may be more convenient. The following is an example BASH script that can be used to set tags based on common selectors in found in the metrics output: collective_name, collective_id, site_name, site_id, appliance_name, appliance_id, func

# Tested on Ubuntu 24.04 with curl and jq installed
set -eEuo pipefail

DD_APP_KEY='< .... >'
DD_API_KEY='< .... >'
DD_SITE='< .... >'

MA='https://appgate-0-ma:5556/metrics'
MA_CA='/etc/datadog-agent/appgate-0-ca.pem'
MA_USERNAME='datadog-agent-0'
MA_PASSWORD='<password goes here>'

# Get a sample of metrics from the metrics aggregator
curl "$MA" --cacert "$MA_CA" --user "$MA_USERNAME:$MA_PASSWORD" \
    --no-progress-meter --fail-with-body |
sed -En 's/apn_status(\{.*}) .*/\1/p' | # Keep only apn_status lines, and only the selectors
sed -En 's/([a-z_]+)=("[^"]*")/"\1":\2/gp' | # Convert '=' to ':' to form a valid JSON object
while read -r sm; do  # For each line
    # do a REST request to Datadog to the host tags endpoint
    curl "https://api.$DD_SITE/api/v1/tags/hosts/$( jq -nr "$sm"'|.appliance_name | @uri' )" \
        --no-progress-meter --fail-with-body \
        --header "DD-APPLICATION-KEY: $DD_APP_KEY" \
        --header "DD-API-KEY: $DD_API_KEY" \
        --json "$( jq -n "$sm"'| { host:.appliance_name, tags:del(.status) | to_entries | map("\(.key):\(.value)") } ' )"
        # JSON request payload has the selectors converted to tags ("key:value") in an array, and the status from apn_status is omitted
done

Please refer to the Datadog API documentation for further guidance.

Metrics

To view the information about the metrics collected by the dashboard, see Datadog’s AppGate SDP Integration documentation.

Viewing the Dashboard

To view the AppGate dashboard:

1. Navigate to the Datadog Marketplace.

2. Search for “appgate” to locate the dashboard.

3. Select the “Appgate SDP” result and follow the steps provided in the Configure tab.

4. Go to the Monitoring Resources tab to select and view the dashboard.

Dashboard Widgets

The AppGate dashboard is made up of widgets grouped into categories. Hovering over or selecting items in each chart or graph will display more information and further options for that item.

Overview

Appliance Status & Usage. Provides a visual of each appliance with the number of hosts represented by hexagons. Usage for each host is represented by the size of the colored icon in each hexagon.

Number of Sites and Appliances by Function. The number of Sites, Gateways, LogServers/Forwarders/Metrics Aggregators, Controllers, Portal/Connectors, and all appliances. For more information, see Sites and About Appliances.

Problems. Displays a list of alerts and warnings and when they were triggered. Monitors have been created to detect issues as they arise.

Sessions. Displays user sessions to appliance functions. The most common type is Gateway sessions, which are used to provide user access to protected networks. Clicking on a line displays more information and options for that appliance.

Gateway

Active Sessions. Displays active sessions over time in a line chart. Clicking on a line displays more information and options for that Gateway.

JS Execution time. Displays JS execution time in seconds in a line chart. JS execution time provides insight into the causes of long event processing time.

Event processing time. Displays event processing time in seconds. Events are generated by user accesses and changes in infrastructure, and need to be processed by Gateways to update user access rules. This chart provides insight into how much time is being consumed by this process.

The Java Heap Usage, Event Queue Usage, and Sessions per Gateway charts display critical metrics related to Gateway load, which can trigger Warning or Error notifications and result in Gateway suspension. When a Gateway is suspended, no new Clients are accepted. For more information about auto-suspend, see Prometheus metrics thresholds.

The Resolutions by Resolver Type and Resolver Cache graphs provide usage and load details of configured resolvers. Resolvers drive powerful name-based Entitlements that are used to control access to dynamic resources. See DNS and name resolution or Cloud API resolvers for more information about resolvers.

Client Reported Metrics. Displays RTT and Connection time in MS by Site.

Firewall Rule Memory. Displays the amount of memory by firewall rule over time.

Sessiond Thread Count. Displays the count of sessiond threads over time.

Controller

Monitors. Displays a list of monitors and their status.

The Authentications, Authorizations, and Scripts Evaluations charts display user auth operations. Errors are displayed in red. These charts display the volume of user activity and the rate of errors.

Database Storage Partition Usage and Java Heap Memory Usage display critical resources for Controllers. Use these metrics to gauge proper sizing of your appliances.

DB Size and DB Replication Lag. These graphs monitor database size and replication lag over time. Non-zero replication lag means that not all Controllers have a consistent view the of the Collective state. This can happen if there is a breakdown in appliance-to-appliance communication. See Controllers for more information.

Audit

Flow. Displays the flow volume of audit logs. Audit logs provide a definitive record of accesses and administrative actions. Forwarding audit logs at high volume can be resource intensive, so be aware of your system’s limits and monitor that the flow is within capacity. See Audit logs for more information.

Client Database Flow IN vs OUT. Audit logs are stored locally until they can be reliably forwarded. A positive value shows a new accumulation of logs, while a negative value shows an emptying of the local database. There can be spikes due to connectivity or forwarder issues.

Drops. A non-zero value displayed here means that the local database was at capacity and could not store new logs.

The Type: Common User Access, Type: Common Administration, and Type: Other graphs display the volume of logs by type. See Audit log detail for more information.

Administration

Next License Expiry In. Displays the amount of days left before then ext license expires. AppGate ZTNA is licensed by users, Portal users, Sites, Connector Resource Groups, and service users. Optional feature licenses will be shown only when they are enabled. See Licenses for more information.

Next Certificate Expiry In. Displays the amount of days left before the appliance certificate expires. AppGate will update the appliance certificate automatically 24 hours before expiration, but this will require a restart of the local services. If the certificate expires, critical appliance operations may stop working. See Internal Certificates for more information.

Certificate Days Remaining. Displays the certificate days remaining before expiry for each appliance.

License Utilization. Displays license information by type, how many are used, and how many are entitled.

Network

The Throughput, Packets, Connections, Service Endpoint Access, and Drops and Errors graphs show load at the network interface level, allowing you to see real-time throughput of user and system activity.

IP Pool Utilization. Displays the allocation and and usage of IP pools. See IPv4 and IPv6 pools for more information.

Single Packet Authorization (New Client and Appliance Connections). Displays single packet authorization over time. Hovering over each vertical bar shows the total number of authorizations at that time.

General

The CPU, Memory, Network, and Storage charts monitor your appliance’s essential resources over time.

Function Status and Sessions. Displays the status of each appliance function and its session activity.