Using AppGate ZTNA telemetry

Prev Next

With the Observability feature license, AppGate ZTNA appliances produce metrics, logs, and traces, and clients produce traces. All of this data uses the OpenTelemetry format. You can send this telemetry into any security information and event management (SIEM) system or observability platform that accepts OpenTelemetry. Your platform choice changes only what questions you can ask once the data arrives.

This page will help you understand telemetry concepts and how to configure a SIEM to receive telemetry data from AppGate ZTNA.

NOTE

Mobile clients do not support telemetry.


OpenTelemetry concepts for AppGate ZTNA

OpenTelemetry is a vendor-neutral format for telemetry. It defines three kinds of data, called signals: metrics, logs, and traces. Each signal answers a different question, and no single signal answers every question.

Using telemetry to troubleshoot AppGate ZTNA

Investigate in three steps: metrics, then logs, then traces. Each step narrows the data and gives you one key for the next step. If you start with logs, you have to read every record by hand.

  1. Find when it happened in the metrics. Find the counter or histogram for the symptom and look for the point where it changes. You leave this step with a narrow time window and a suspect appliance.

  2. Find what happened and to whom in the logs. Filter the logs from that time window to one service. Group by an attribute to rank the causes. You leave this step with a failure reason, a user, and often a trace_id.

  3. Find where the time went in the trace. Open the trace and look for spans or activities that take longer time than expected.

NOTE

AppGate ZTNA uses tail sampling for traces. The Telemetry Aggregator waits until a trace is complete and then decides whether to keep it. It keeps every trace that contains an error and 10% of successful client traces. If you cannot find the trace for a request that succeeded, the Telemetry Aggregator likely discarded it to reduce traffic and SIEM storage. Use metrics and logs to investigate that request.

Signal

Answers

Example

Metrics

When did it happen?

Sign-in latency rose at a specific time.

Logs

What happened, and to whom?

The Gateway denied a named user access to a destination.

Traces

Where was the time spent?

One Site sign-in accounted for most of a slow sign-in.

Flowchart illustrating metrics, logs, and traces for analyzing system performance.

Metrics

A metric is a number sampled on a schedule, usually every 15 seconds. Examples include apn_cpu_usage_percent and ctr_client_average_time.

Metrics cost little to store and query across months, because each sample is a number with a few labels. The trade-off is that a metric combines all users into one value. A metric shows that sign-in slowed down at a specific time, but it cannot show which users the slowdown affected.

Metrics combine users because of cardinality, the number of distinct label combinations. A metric labeled by Site has few values. A metric labeled by user has thousands, and the backend platform stores each combination as a separate series. Labeling a metric with a user identifier can overload a metrics platform, so keep user-level detail in logs and traces.

Metrics currently carry no link to logs or traces; you must correlate them by time window and appliance only.

See Prometheus metric details for information about the metrics collected in AppGate ZTNA.

Logs

A service writes a log record when an event occurs. It carries detailed information that is too specific for a metric, such as the user distinguished name, destination IP address, and entitlement token ID.

OpenTelemetry log records are structured. Each record has a short body that names the event and a set of typed attributes. The following example shows a packet access record from a Gateway:

body                                                packet ACL access record
service_name                                        cz-vpnd@1
service_namespace                                   appgate.gateway
host_name                                           <gateway-hostname>
severity                                            INFO
appgate_gateway_vpnd_ip_access_action               allow
appgate_gateway_vpnd_ip_access_protocol             UDP
appgate_gateway_vpnd_ip_access_source_ip            192.0.2.30
appgate_gateway_vpnd_ip_access_destination_ip       198.51.100.207
appgate_gateway_vpnd_ip_access_destination_port     54113
appgate_gateway_vpnd_ip_access_distinguished_name   CN=<user-id>,CN=...
appgate_gateway_vpnd_ip_access_entitlement_token_id <token-id>
telemetry_sdk_name                                  opentelemetry

Because appgate_gateway_vpnd_ip_access_action is a field, you can count denied packets by protocol with one grouped query instead of a regular expression.

Logs show what happened and to whom. They do not record duration, so they cannot show where the time went. Logs are grouped by service_name (the daemon) and service_namespace. To know which kind of appliance the service_name refers to, see Daemon names in logs.

Traces and spans

A trace follows one request across every component that handles it. A trace consists of spans. Each span is one timed operation with a name, start time, end time, parent, and attributes.

Spans nest. For example, the client sign-in span is the parent of the HTTP call to the Controller. That call is the parent of the token creation inside the Controller. A SIEM or observability platform draws nested spans as bars on a shared time axis, in a view called a waterfall. To find where a slow request spent its time, look for the widest bar whose child spans do not account for its width.

Every span in a trace carries the same trace_id. Log records written during a span can carry the same trace_id, which lets you move between a log record and its trace. AppGate ZTNA does not export every trace; see the note in Using telemetry to troubleshoot AppGate ZTNA for how traces are sampled

Waterfall trace diagram highlights a timing gap between token creation and the same-trace log record.

See AppGate telemetry spans for spans and their descriptions.


Configuring a SIEM or observability platform for AppGate ZTNA

Configure your SIEM or observability platform to receive all three signals and set up your Telemetry Aggregator. These steps apply to any SIEM or platform that accepts OpenTelemetry. For product-specific settings, see your vendor documentation.

Prerequisites

Before you configure the SIEM or observability platform, confirm the following:

  • The SIEM or platform accepts OTLP for logs, traces, and metrics.

  • Each Telemetry Aggregator appliance in the Collective can reach the SIEM or platform OTLP endpoint over the network, and all other appliances can reach the Telemetry Aggregator(s).

  • You have administrator access to the SIEM or platform and to the AppGate ZTNA admin UI.

Configuration

  1. On your SIEM or observability platform, enable OTLP ingestion for logs, traces, and metrics. The OpenTelemetry default ports are 4317 for gRPC and 4318 for HTTP. Confirm the ports your SIEM or platform uses.

  2. In the AppGate ZTNA admin UI, configure an OpenTelemetry Protocol Exporter or Datadog Exporter on the Telemetry Aggregator to send to <backend-otlp-endpoint> in the Endpoint field.