LogServer migrations

Migrating LogServer to another appliance

If you have your LogServer co-hosted with a Controller it is a good idea to move it to a separate appliance. You can migrate your LogServer away from your Controller taking the existing audit logs across. Appliances will normally queue logs when they can't talk to the LogServer; but they only do this when there is a LogServer configured. So it is important to minimize the time window while there are no LogServers configured.

You will need a browser access the admin UI and terminal access (ssh) to both Appliances:

1. in the browser, open the admin UI and login as administrator

2. go to Appliances and create a new appliance (B) which will become the new LogServer

3. go to Appliances > Appliance (A) (the appliance that is currently acting as LogServer) and disable the LogServer on the appliance, i.e. uncheck LogServer and press Save

4. in a terminal, ssh to the Appliance (A) were the LogServer is currently running: ssh cz@applianceA.address

   1. switch user to root: sudo su  

   2. go to the Open search directory: cd /mnt/data/elasticsearch5

   3. compress the whole Open search data directory: tar zcvf es_data.tar.gz data/

   4. copy the compressed data file to the new appliance: scp es_data.tar.gz cz@applianceB.address:

5. in a terminal, ssh to the new spare Appliance (B): ssh cz@applianceB.address

   1. switch user to root: sudo su  

   2. copy the compressed file to its proper directory: mv es_data.tar.gz /mnt/data/elasticsearch5

   3. go to the Open search directory and uncompress the data file: cd /mnt/data/elasticsearch5; tar zxvf es_data.tar.gz

   4. set the right permissions to the files: chown -R czd:czd *

6. in the admin UI, go to Appliances>Appliance (B) and enable the LogServer, i.e. check LogServer and press Save

Migrating from using LogServer to using LogForwarder

If you have been using the LogServer it might now be time to migrate to an enterprise log management system. The LogForwarder is designed for this purpose.

LogForwarders can be configured for HA operation using two or more appliances. They can be deployed to export the logs by Site to different destinations. Multiple export protocols can be specified at the same time including one for the ELK stack. This means that if there is an ongoing requirement to retain the ELK stack (effectively a copy of the LogServer) in an enterprise environment then one can be deployed outside of the AppGate ZTNA Collective (for example, running in AWS) and the logs forwarded there whilst also exporting the log data into an enterprise-class logging system.  

It is possible to migrate from LogServer to use the LogForwarder without loosing the audit logs you have already accumulated on you LogServer. To do this:

1. Disable the LogServer function and enable the LogForwarder function on the LogServer. This will become a LogForwarder.

2. SSH to the LogForwarder (old LogServer) and run command to generate Opensearch configuration: sudo cz-config set -j runOpensearch true

3. Wait for Opensearch to become green and this can take a very long time depending on the size of the database. To see this run:

   es-tools -c cluster-health

   {
     "cluster_name": "appgate",
     "status": "green",
     "timed_out": false,
     "number_of_nodes": 1,
     "number_of_data_nodes": 1,
     "active_primary_shards": 6,
     "active_shards": 6,
     "relocating_shards": 0,
     "initializing_shards": 0,
     "unassigned_shards": 0,
     "delayed_unassigned_shards": 0,
     "number_of_pending_tasks": 0,
     "number_of_in_flight_fetch": 0,
     "task_max_waiting_in_queue_millis": 0,
     "active_shards_percent_as_number": 100
   }

4. Next, es-export is used to export all the log indexes from Opensearch and feed them to cz-logd just like any other appliance. Run es-export script with the custom forward parameter:

   es-export --forward_logd true

   Current index:  logstash-2021-02-23
   Done

5. Finally, to stop Open search and remove the temporary Open search configuraiton on the new LogForwarder, do: sudo cz-config set -j runOpensearch false

6. Optionally, remove the Open search database (no going back): sudo rm -rf /mnt/data/elasticsearch5/data/