When resource names are used to define the (protected) Host(s) it becomes possible to automate the configuration within Entitlements. This allows the network resources to be populated by external systems - something which is much more suitable for today’s fast changing dev-ops world.
The Appgate SDP system makes API calls that return the actual instances currently active in a specific Cloud environment. This allows the firewall rules to be populated in near real-time and avoids any need to manually update Entitlements (firewall rules) every time a new server instance is created or removed.
Using resource names
If for example the resource name azure://{"virtual-networks":"example"} was used, the Gateway will query the appropriate provider (in this case Azure) to get the current IP addresses for all the network resources in the vNet Example. The resolvers work recursively, so if this first query does not return IP addresses (as might be the case when querying a Load Balancer) then the process is repeated. Once it returns IPs, these will be used to set the required Client routes and firewall rules. When a new host is added to the vNet Example, it will automatically be added to the results the next time the provider is queried for information (default time is set to 60s). Support is provided for the following Cloud environments:
Syntax | Used for |
|---|---|
aws:// | Amazon Web Services |
azure:// | Microsoft Azure |
gcp:// | Google Cloud Platform |
illumio:// | Illumio |
esx:// | VMware vSphere |
Depending on the Cloud provider you may need to configure the hosting environment and Appgate SDP:
Within the hosting environment (such as AWS), use tags and security groups to identify the instances, network interfaces, or load balancers.
Set permissions in, or obtain credentials from, the hosting environment that grants the API user the rights to use all relevant REST API calls.
Set up the different types of cloud resolvers in Sites > Cloud Resolvers so they are ready to be used by the Gateway(s).
Define the Host(s) in Entitlements > Actions using the special resource name syntax
You can test your name resolver syntax in the Sites page. There is a Test Name Resolver action button for each site. The syntax for testing the resolver will be exactly the same as the syntax that you configure in your actions.
Refer to Cloud API resolvers for more details.
All Cloud providers have a very strict rate limit at which you can make API calls, therefore instead of making an API call for every name the Appgate SDP name resolver makes a few API calls at defined intervals to download all available resources. For example in AWS all instances, load-balancers, security groups, etc will be downloaded, now when a name needs resolving it will be cached locally instead of making an API call. Say you have 1000 different resource names specified (in the users' Entitlements) and you have 6 AWS accounts across 3 regions; then this might require the Gateway to make 18,000 individual API calls. Instead, it only needs to make 18 API calls (once every update interval) to download all the results for each account in all the regions. Because the results are cached in memory the look-up occurs locally yielding instant results. This avoids waiting for API calls to get the results and avoids any likelihood of hitting the Cloud providers' rate limit.
The name resolvers all use a common JSON syntax. This is much more flexible, allowing complex queries to be constructed, and avoiding the need for cumbersome random operators and escape characters. It is also supports simpler automation when writing javascript for use in Entitlement scripts.
Common JSON syntax
Common JSON syntax used throughout however there are significant differences in the implementation for each resolver. So please be sure to refer to the each resolver and not assume that something from one type can be copied directly to another type.
String VALUEs within " " may contain spaces; e.g. ["This That"].
A JSON object contains one or more NAME:VALUE pairs enclosed in { }.
Some object types allow multiple comma separated NAME:VALUE pairs (tags, security groups etc.). When this is the case all VALUES must be present (logical AND): {"type":"red", "another-type":"large"}
Where appropriate multiple ,(comma) separated objects can be specified, in which case any VALUE can be present (logical OR): {"type":"red", "another-type":"large"}, {"instance":"This"}, {"instance":"That}
NOTE
The existing (prior to v6.3) legacy syntax unique to each resolver can still be used. All resource names defined with legacy syntax will now be dynamically interpreted to the common JSON syntax before resolution takes place.
Amazon Web Services (AWS) resolver
Supports resolving names from AWS using the AWS SDK for Go v2. To use a name resolver with the AWS API requires having a configured IAM role with appropriate permissions or credentials in the form of an Access Key ID with a respective Secret Access Key.
Resolver supports the following syntaxes:
Common JSON syntax | Legacy syntax | Description |
|---|---|---|
aws://{"instances-security-groups":["VALUE"]} | aws://security-group:<VALUE> | Resolves to the IP addresses of all instances with security groupName VALUE. |
aws://{"instances-tags":[{"tag-key":"KEY","tag-values":["VALUE"]}]} | aws://tag:<KEY>=<VALUE> | Resolves to the IP addresses of the instances with a tag key KEY and tag value VALUE. |
aws://{"lbs":["VALUE"]} | aws://lb:<VALUES> | Resolves to the hostnames with LoadBalancerName VALUE. |
aws://{"lbs-security-groups":["VALUE"]} | aws://lb-security-group:<VALUE> | Resolves to the hostnames of the load-balancers with security GroupName VALUE. |
aws://{"lbs-tags":[{"tag-key":"KEY","tag-values":["VALUE"]}]} | aws://lb-tag:<KEY>=<VALUE> | Resolves to the hostnames of the load-balancers with a tag key KEY and tag value VALUE. |
aws://{"lbsv2":["VALUE"]} | aws://lbv2:<VALUE> | Resolves to the hostnames of the load-balancers with LoadBalancerName VALUE. |
aws://{"lbsv2-security-groups":["VALUE"]} | aws://lbv2-security-group:<VALUE> | Resolves to the hostnames of the load-balancers v2 with security GroupName VALUE. |
aws://{"lbsv2-tags":[{"tag-key":"KEY","tag-values":["VALUE"]}]} | aws://lbv2-tag:<KEY>=<VALUE> | Resolves to the hostnames of the load-balancers v2 with a tag key KEY and tag value VALUE. |
aws://{"enis-security-groups":["VALUE"]} | aws://eni-security-group-name:<VALUE> | Resolves to the IP addresses of the network interfaces with security GroupName VALUE. |
aws://{"enis-security-groups-ids":["VALUE"]} | aws://eni-security-group:<VALUE> | Resolves to the IP addresses of the network interfaces with security GroupId VALUE. |
aws://{"enis-tags":[{"tag-key":"KEY","tag-values":["VALUE"]}]} | aws://eni-tag:<KEY>=<VALUE> | Resolves to the IP addresses of the network interfaces with a tag key KEY and tag value VALUE. |
aws://{"terms":[{"instances-security-groups":["VALUE"]},{"lbs":["*"]}]} | - | Resolves to the IP addresses of all instances in security group VALUE + resolves to the IP addresses of the load-balancer VALUE |
- | aws://eni:<VALUES> | In AWS the network interfaces do not have any real naming attribute. They have tags and when you add a descriptive Name it basically adds a new tag with key Name and value the value given. enis-tags with "tag-key":"Name" should be used instead to be explicit rather than assuming anything |
aws://{"rds-db-instances":["VALUE"]} |
| Resolves to the hostnames of all instances with the RDS DB instance groupName VALUE. |
aws://{"rds-db-instances-tags":[{"tag-key":"KEY","tag-values":["VALUE"]}]} |
| Resolves to the hostnames of the RDS DB instances with a tag key KEY and tag value VALUE. |
aws://{"eks-cluster-tags":[{"tag-key":"KEY","tag-values":["VALUE"]}]} |
| Resolves to the hostnames of the EKS clusters with a tag key KEY and tag value VALUE. |
aws://{"eks-clusters":["VALUE"], "public-ips":true} |
| Resolves to the hostname for the EKS clusters and if the IPs are public. |
aws://{"eks-clusters":["VALUE"], "account": "VALUE", "region": "VALUE", "public-ips":true} |
| Resolves to the hostname of the EKS clusters, specifies the AWS account and region, and if the Ips are public. |
Note the following:
These formats for the API queries are based on a subset of the published Amazon EC2 filters.
When hostnames are returned by the Cloud AP (i.e. load-balancers) then these are resolved using the appliances' DNS server(s).
Both syntaxes: supports glob expressions for VALUE; e.g.
aws://{"instances-tags":[{"tag-key":"KEY","tag-values":["foo\\*boo"]}]}.Common JSON syntax: supports glob expressions for KEY; e.g.
aws://{"instances-tags":[{"tag-key":"foo\\*boo","tag-values":["VALUE"]}]}.Common JSON syntax: supports multi terms. You can define the terms as a JSON list:
aws://{"terms":[term1, termN]}. It is a compact equivalent of declaring multiple different names. You can query for many different resources and resources' attributes (tags, security groups etc.) within one resource name and you can apply different options to different terms e.g.aws://{"terms":[{"lbsv2":["foo*"], "regions":["europe*"]},{"lbsv2-security-groups":["default"]},{"lbsv2-tags":[{"tag-key":"Deployment","tag-values":["red"]}]}]}.In this example we query for load-balancer(s) v2 which their name has suffix foo only in regions which have prefix europe or they belong to the security group default or they have tag key Deployment with tag value red.Common JSON syntax: for OR operations add more comma separated values in the JSON lists e.g.
aws://{"lbs":["VALUE1", "VALUE2"]}.Legacy syntax: for load-balancers both private and public IPs will be returned.
Legacy syntax: for OR operations you can use the character | e.g.
aws://lb:<VALUE1>|<VALUE2>.Legacy syntax: there may be occasions where the characters &, | are used within values which would conflict with their use within the resolver syntax. There might for example be an AWS instance tag named serversE&W. This would require escaping with the special character \ e.g.
aws://tag:serversE\&WFor all kinds of tags:
Common JSON syntax: for AND operations add more tag objects in the JSON lists e.g.
aws://{"instances-tags":[{"tag-key":"Name","tag-values":["*"]},{"tag-key":"environment_id","tag-values":["*"]}]}.Legacy syntax: for AND operations you can use the character & e.g.
aws://tag:Name=*&environment_id=*.
Extra options
Public IP addresses (the resolver normally returns the private IP address of the resource(s)):
Common JSON syntax: add "
public-ips":true to your query; e.g.aws://{"instances-security-groups":["*"], "public-ips":true}.Legacy syntax: append &publicIp to the end of your query; e.g.
aws://security-group:*&publicIp.NOTE:By default the UI will add the option for the public IP address when configuring new load-balancers. If the internal IP is required just set this to false.
Regions/Accounts: By default the resolver will search all configured accounts and regions. When filtering by accounts you can use: the account's ID, or the account ID's of the assumed roles, or if you use "master" this refers to the account in which the Gateway is running. If you only want a specific account or region, then:
Common JSON syntax: add values to the JSON list(s)
"regions":[ ],"accounts":[ ]for the appropriate term(s); e.g.aws://{"instances-security-groups":["*"], "accounts":["master"],"regions":["eu-west-2"]}.Legacy syntax: use ®ion=<VALUES>&account=<VALUES>; e.g. aws://security-group:*®ion=eu-west-2&account=master.
Example
Common JSON syntax | Description |
|---|---|
aws://{"terms":[{"lbsv2":["foo*"], "regions":["europe*"]},{"lbsv2-security-groups":["default"]},{"lbsv2-tags":[{"tag-key":"Deployment","tag-values":["red"]}]}]} | Resolves to the private IP addresses of load-balancer(s) v2 with their name suffix foo only in regions which have prefix europe or they belong to the security group default or they have tag key Deployment with tag value red. |
Microsoft Azure resolver
Supports resolving names from Azure using the Azure SDK for Go. To use a name resolver with the Azure API it is required that you have either a configured Azure AD App with reader rights to the resource groups this resolver should have access to (an Azure AD App can be created with the azure cli command azure ad app create). Alternatively, the Gateway must be instantiated within Azure itself with the appropriate Managed Identity configured.
Resolver supports the following syntaxes:
Common JSON syntax | Legacy syntax | Description |
|---|---|---|
azure://{"network-security-groups":"VALUE"} | azure://network-security-group:<VALUE> | Resolves to the IP addresses of all instances in security group VALUE |
azure://{"network-security-groups-tags":{"tag-key":"KEY", "tag-value":"VALUE"}} | azure://nsg-tag:<KEY>=<VALUE> | Resolves to the IP addresses of the network security groups with a tag name KEY and tag value VALUE |
azure://{"virtual-machines-tags":{"tag-key":"KEY", "tag-value":"VALUE"}} | azure://tag:<KEY>=<VALUE> | Resolves to the IP addresses of the instances with a tag name KEY and tag value VALUE |
azure://{"lbs":"VALUE"} | azure://lb:<VALUE> | Resolves to the IP addresses of the load-balancer with a name VALUE |
azure://{"lbs-tags":{"tag-key":"KEY", "tag-value":"VALUE"}} | azure://lb-tag:<KEY>=<VALUE> | Resolves to the IP addresses of the load-balancers with a tag name KEY and tag value VALUE |
azure://{"virtual-networks":"VALUE"} | azure://vnet:<VALUE> | Resolves to the IP addresses (the entire Vnet) of all instances in virtual network VALUE |
azure://{"subnets":"VALUE"} | azure://subnet:<VALUE> | Resolves to the IP addresses (the entire Subnet) of all instances in subnet VALUE |
NOTE
Both syntaxes: support glob expressions for KEY and VALUE; e.g. azure://{"network-security-groups-tags":{"tag-key":"ko\\*\\*smas\\[", "tag-value":"boo"}}.
In this example the tag-key to be matched is ko**smas[ . The asterisks * and the brackets [ have special meanings in glob expressions, so they need to be escaped with \ and because the glob expression is inside JSON a second \ is needed.Legacy syntax: supports spaces in KEY and VALUE; e.g. azure://tag:Application=Software Defined Perimeter
Extra options
Public IP addresses (the resolver normally returns the private IP address of the resource(s)):
Common JSON syntax: add "public-ips":true to your query; e.g. azure://{"network-security-groups":"*", "public-ips":true}.
Legacy syntax: does not support public IP addresses.
NOTE
By default the UI will add the option for the public IP address when configuring new load-balancers. If the internal IP is required just set this to false.
Google Cloud Platform (GCP) resolver
Supports resolving names from GCP using the Google SDK for Go. To use a name resolver with the GCP API requires having a configured IAM role. GCP's filtering is basically a scripting language therefore "code" needs to be given to the GCP engine for evaluation. However as the rate limit for the resources filtering API calls is low, especially for projects, Appgate SDP has implemented a local parsing engine for the scripting language so that it can download all the resources once per defined interval and then filter them locally. As a fallback, in the situation where the local parsing engine produces an error, the query will be passed to the GCP engine through an API call and a warning sent to the admin feed; the local parsing engine might for instance fail if some newly added attribute is queried.
NOTE
In the GCP API there is no direct object for Load Balancers therefore the Forwarding Rules filter is used to query for Load Balancers.
Site level filters
In System > Sites there are fields for Projects filter, Instances filter and Forwarding Rules filter - these can normally be left blank. If these fields are used, any expressions defined there will be ANDed to any expressions defined in the Entitlement Actions. This can be useful to limit the scope of a query in large GCP deployments. But, When troubleshooting name resolvers, remember to check the Site settings as the problem might lie there, and not in your resource name expression.
NOTE
In the situation where no Projects filter has been defined (neither in the Site nor in the Entitlement Action), then the project ID in which the Appliance is running will be assumed.
Resolver supports the following syntaxes
Common JSON syntax | Legacy syntax | Description |
|---|---|---|
gcp://{"projects-filter": "GCP syntax"} | gcp://project=<GCP syntax> | The Projects filter syntax is defined in the GCP projects search |
gcp://{"instances-filter": "GCP syntax"} | gcp://instance=<GCP syntax> | The Instances filter syntax is defined in the GCP instances aggregated list |
gcp://{"forwarding-rules-filter": "GCP syntax"} | - | The Forwarding Rules filter syntax is defined in the GCP forwarding rules aggregated list |
gcp://{"instances-network-tags": "VALUE"} | gcp://tag=<VALUE> | The Instances Network Tags filter is provided by Appgate SDP. |
gcp://{"service-account-emails": "VALUE"} | gcp://serviceAccountEmail=<VALUE> | The Services Account Emails filter is provided by Appgate SDP. |
Note the following:
GCP's regexp which is called Google RE2 syntax cannot be used with logical operators, so
gcp://{"instances-filter": "-labels.foo:* AND zone eq .*europe-west1-.*"}is not allowed in the RE2 syntax. However Appgate SDP's local parsing engine (for RE2 syntax) does support the use of AND as a logical operator. so any Site level filters can be ANDed in the usual way. The use of AND as a logical operator is not recommended; should an error in the local parsing engine causes the the query to be passed to the GCP engine, this will also return an error (because of the AND) - so the query will fail.Both syntaxes: supports glob expressions for service-account-emails and instances-network-tags: e.g.
gcp://{"instances-filter": "name:purple*", "service-account-emails": "appgate-service-account@appgate-dev.iam.?se[r-z]viceaccount.?om"}.Legacy syntax: requires that any filter strings containing spaces must be quoted using ' and any quoted strings that use ' that are inside a filter expression already using ' should be escaped using \ e.g.
gcp://project=id:appgate-dev;instance='cpuPlatform:\'Intel Broadwell\' AND scheduling.automaticRestart:true'
Extra options
Public IP addresses (the resolver normally returns the private IP address of the resource(s)):
Common JSON syntax: add "public-ips":true; e.g.
gcp://{"instances-filter": "-labels.foo:*", "public-ips":true}.Legacy syntax: append ;natIP to the end of your query; e.g.
gcp://instance=-labels.foo:*;natIP
Magic zone variable: In GCP, the zones are expressed with a URL path e.g.
projects/appgate-dev/zones/europe-west3-cand in order to query for them with the GCP syntax you need to either consider the full URL e.g.https://www.googleapis.com/compute/v1/projects/appgate-dev/zones/europe-west3-cor use the "GCP's regexp" called Google RE2 syntax e.g.gcp://{"instances-filter": "zone eq .*europe-west3-.*"}.The RE2 syntax has the regexp limitation mentioned above, so to provide more flexibility, Appgate SDP accepts a special variable${ZONE_PROJECT_URL}(or for the regions of the forwarding rules use${REGION_PROJECT_URL})which when used, will automatically construct the correct URL for the local parsing engine. To filter the query by zone (or region) use:Common JSON Syntax:
gcp://{"projects-filter": "id:appgate-dev", "instances-filter": "-labels.foo:* AND zone:\"${ZONE_PROJECT_URL}europe-*\""}.Legacy syntax:
gcp://project=id:appgate-dev;instance='-labels.foo:* zone: "${ZONE_PROJECT_URL}europe-*"'.
Examples
Here are some typical queries showcasing the usage of the GCP syntax within Appgate SDP. For more information about the GCP language refer to the documentation linked in the Syntax section above. Remember that any Site level filters will be ANDed respectively to the filters of the names thus modifying them and in that case the native Google RE2 syntax will not be used.
Common JSON syntax | Legacy syntax | Description |
|---|---|---|
|
| Resolves to the private IPs from instances with name that has prefix purple |
|
| Resolves to the private IPs from instances that have a label with key foo and value bar |
|
| Resolves to the private IPs from instances that have a label foo. Projects to look for must have id with value appgate-dev |
|
| Resolves to the private IPs from instances that do not have a label foo. Projects to look for must have id with value appgate-dev |
|
| Resolves to the publics IPs from instances that do not have a label foo and their Service Account Email is appgate-service-account@appgate-dev.iam.gserviceaccount.com. Projects to look for must have id with value appgate-dev |
|
| Resolves to the private IPs from instances with CPU Platform Intel Broadwell and automatic restart true. Projects to look for must have id with value appgate-dev |
|
| Resolves to the private IPs from instances in zones that are prefixed with europe-. Projects to look for must have id with value appgate-dev |
|
| Using GCP's regexp (logical operators not allowed): resolves to the private IPs from instances in zones that are prefixed with europe- .Projects to look for must have id with value appgate-dev |
| - | Resolves to the public IPs from forwarding rules with name that has prefix testlb and the port range 80-80 |
| - | Resolves to the public IPs from forwarding rules with target that has substring testlbhttp-target-proxy.
|
| - | Resolves to the public IPs from forwarding rules with backendService that has substring testlbtcp.
|
| - | Resolves to the public IPs from forwarding rules in zones that are prefixed with europe-.
|
| - | Using GCP's regexp (logical operators not allowed): resolves to the public IPs from forwarding rules in zones that are prefixed with europe- .
|
Illumio resolver
Supports resolving names from the Illumio platform using its REST API. To use a name resolver with the Illumio API it requires credentials with reader rights.
Resolver supports the following syntax:
Common JSON Syntax | Description |
|---|---|
| Resolves to the private IP addresses of virtual machines with label-values containing (VALUE1 and VALUE2) or (VALUE3). |
NOTE
Common JSON syntax can include a list of lists. Each list could instead be added as a new Action. Within a single list ALL labels must be present to be a match.
Extra options
Public IP addresses: By default the resolver will return the private IP address of the resources. If you want only public IP addresses returned, then add "public-ips":true If you want to return both public and private IP addresses, add a second Illumio target to your entitlement with "public-ips":true
VMware vSphere resolver
Supports resolving names from VMware vCenter using the Vmware SDK for Go. To use a name resolver with the VMware API requires credentials with reader rights.
Resolver supports the following syntaxes:
Common JSON syntax | Legacy syntax | Description |
|---|---|---|
esx://{"virtual-machines":"VALUE"} | esx://vm:<VALUE> | Resolves to the private IP addresses of virtual machines with name VALUE. |
esx://{"folders":"VALUE"} | esx://folder:<VALUE> | Resolves to the private IP addresses of all virtual machine's in the folder with name VALUE. Common JSON syntax supports absolute path esx://{"folders":"foo/bar"} |
esx://{"resources-pools":"VALUE"} | esx://resourcepool:<VALUE> | Resolves to the private IP addresses of all the virtual machine's in the resource pool with name VALUE |
esx://{"tags":"VALUE"} | esx://tag:<VALUE> | Resolves to the private IP addresses of a virtual machine or Folder or Resource pool with tag name: VALUE |
NOTE
Both syntaxes: supports glob expressions for VALUE; e.g. esx://{"folders":"purple*"}.
Legacy syntax: supports spaces in VALUE; e.g. esx://folder:Software Defined Perimeter