Notifications

562 views

Details

This KB article has some general info on how MID Servers are affected by Instance Clones, and some of the things you need to keep in mind.

Are MID Servers Cloned?

No. When a production instance is cloned over a sub-production instance, the production instance's MID Servers are not duplicated or copied, and the sub-production instance will not end up with the same MID Servers as production.

MID Server installations will always point to the same single instance url set in the config.xml file of that installation, regardless of whether that instance is now actually a copy of another instance.

You will need at least one MID Server installation for each instance. You can install multiple MID Servers on the same host server to accomplish this, so long as they have different MID Server names.

In order to be able to effectively test MID Server features in sub-production instances, it is recommended to mirror the production MID Server configurations and install the MID Servers within the same network environments.

The MID Server's login credential

After a clone has overwritten a target instance with a new configuration, data and even a different version, existing MID Servers for that target instance will still be trying to connect to it using the same username and password that was originally configured for the MID Server.

The user table in the instance will now be a copy of the source instance's user table, and so the MID Server may be trying to log in with a user that now doesn't exist in the instance or may have a different password. That behavior is controlled by the "Preserve users and related tables" checkbox in the options when requesting a clone.

To avoid this possibility, the recommendation is to use the same username and password for MID Servers on all instances. Different MID Servers for different functions can have different passwords, but the MID Servers for a particular function (e.g. Discovery of a particular region/datacenter) would use the same username/password.

For more information, see: Active MID Server post-cloning credential issues

The Instance Credentials table

Prior to the New York release, clones would replace the Credentials table [discovery_credentials] with the records from the source of the clone. Since New York, this table is both Excluded and Preserved in the clone, so that the sub-production instance keeps the Credential records it used to have before the clone.

This table is used by Discovery/Service Mapping probes from the MID Server, but also some other recent integrations features that may not even use the MID Server.

It is possible to configure this behavior yourself, if you do not want to use the default settings. For more information, see:
Exclude a table from cloning
Data preservation on cloning target instances

Note: These excludes/preservers, including the out-of-the-box ones, only take effect if the "Exclude tables specified in Exclusion List" and "Exclude audit and log data" options are checked when requesting the clone.

There is a problem with Data Preserver and Exclude code in the clone engine, where it doesn't automatically preserve all child tables of extended tables. This breaks the OOTB Exclude setting in New York for discovery_credentials, and also custom settings, leaving corrupt credential records on the target. For more information and a workaround, see:
KB0717208/PRB1305469 Excluding table-per-class (TPC) extended tables from a clone can cause orphaned Discovery Credentials with the 'Record not found' error when trying to open them
That was fixed, then broken again by PRB1403259. A more recent problem PRB1391898 causes the same corruption, related to the preservers rather than excludes.

If the source and target instance have different plugins installed, then you can also get corrupt credential records. For example:

  • You install Cloud Management on the sub-prod instance, which adds extended tables to the Discovery Credentials tables specific to Cloud functionality. e.g. sn_cmp_ssh_credentials. 
  • You create a sn_cmp_ssh_credentials credential. Being an extended table, that means you have the same sys_id in both discovery_credentials and sn_cmp_ssh_credentials tables.
  • You clone from production, which does not have Cloud Management installed, and so the sub-prod instance instance will now not have the Cloud Management plugin any more, or it's tables.
  • The default excludes/preservers mean the discovery_credentials table on the dev instance is preserved. The record with sys_class_name=sn_cmp_ssh_credentials still exists in the dev instance, but the sn_cmp_ssh_credentials table does not any more, meaning the corrupt record can't be opened, edited, deleted, and may also break MID Servers trying to load credentials. 

To avoid breaking those records, you will need to install the same plugin(s) on production before the clone, or delete those credential records before the clone. If you have broken records because of this cause, installing the plugin after won't help, because you will still be missing the sys_id from the child table, and the record will remain corrupt. It will probably need deleting using a Table Cleanup [sys_auto_flush] job, as that deletes at a lower level that lists and forms. That method is describes in:
KB0723549 How to repair Discovery Credentials not accessible after clone

The MID Server's own records

A MID Server installation can only be used by one instance. Production MID Servers will remain pointing to production, and so any records for those MID Servers don't need to be copied, and so are excluded from the clone. MID Servers for the sub-production instance will still be pointing to that, so records for those MID Servers need preserving.

This includes related records such as Properties, Parameters, Clusters, Capabilities, IP Ranges, Applications, Issues and records for the MID Server Dashboard.

Warning: Prior to Madrid, there was a known problem which means this isn't done properly. Upgraded instances may also not get the fixed exclude/preserve lists, and may need fixing manually. For more information, see:
PRB1287729 / KB0688954 - The MID Server Exclude/Preserve Clone settings cause MID Server issues on the target

MID Server Properties is a difficult table, because some are for all MID Servers, but others are for specific MID Servers. A clone will delete and overwrite those records. Any specific MID Server with non-default MID Server Properties may need manually re-configuring after a clone (which is PRB1388744).

There are some ecc_agent... tables which should never be excluded in clones, because they contain Code, which is also version specific. Excluding these can cause missing or mismatched code after a clone, and the MID Server won't work as expected.

  • ecc_agent_jar
  • ecc_agent_mib
  • ecc_agent_script
  • ecc_agent_script_file
  • ecc_agent_script_include
  • ecc_agent_script_param
  • and others..

Note: These excludes/preservers, including the out-of-the-box ones, only take effect if the "Exclude tables specified in Exclusion List" and "Exclude audit and log data" options are checked when requesting the clone.

Records and settings that reference MID Servers

In settings of all features that use MID Servers, there are likely to be references to specific MID server Sys IDs. e.g.

  • Discovery Schedules set to use specific MID Servers or MID Server Clusters
  • Import Data Sources set to use specific MID Servers
  • Credential records restricted to specific MID Servers
  • etc.

A lot of those settings will have been copied over with the clone, but the MID Servers won't exist on the target instance, and so many of those features will now not work or do something unexpected and need re-configuring to use the target instance's MID Servers.

Post-Clone Cleanup Scripts can be used to fix or prevent a lot of these settings after a clone. e.g.
KB0789119 A Post-Clone script to Deactivate and Cancel Discovery Schedules

It is possible to configure MID Servers to have the same sys_id as the production MID Servers. If the installations are on the same host server, the names need to be different, but the sys_ids can be the same. This process and the pros and cons are explained in more detail in:
KB0719301 How to manually Clone a MID Server so that you don't have to reconfigure all integrations features to use a different MID Server on the sub-prod instance after an Instance Clone

Warning: Prior to New York, Discovery Schedules set to specific production MID Servers may unexpectedly run after clones:
PRB1311068 / KB0788922 After a Clone, a Discovery Schedule for a Specific MID Server, which does not exist in the target instance and so is now a broken reference, will still run in a random MID Server

Clones that change the Version of the instance

After a clone, existing sub-prod MID Servers will suddenly find themselves communicating with a 'different' instance compared to the previous incarnation of that instance. It will still try to connect, and will then try to upgrade/downgrade to match the version the instance is now.

Upgrades should work, and plenty of regression testing is done every release to make sure it does.

However it is possible to have a situation when the instance ends up on an earlier version (e.g. a sub-prod instance is used for an upgrade test, and then cloned over again), and then the MID Server may have problems. A MID Server running a future version compared to the instance isn't going to be able to communicate with the instance properly using API and encryption/validation features that are only present in the future instance version.

A downgrade from a version that uses Signed ZIP files to one that doesn't will require a MID Server restart before the downgrade will work:
MID Server fails to upgrade from a Signed ZIP file version to a Non-Signed version, because instanceinfo is cached. e.g. MP10 HF1b to NP8, or NP9 to OP2

In a Downgrade situation, especially between major versions, you are going to need to be prepared to manually re-install the MID Server. For more details of that repair process see:
KB0713557 How to manually Upgrade and/or Restore a MID server after a failed Upgrade

Agent Client Connector

Each Agent Client Collector installation communicates to a MID Web Server Extension running on a specific MID Server., and therefore specific instance name  The ACC Websocket Endpoint and MID Web Server Contexts need to still be there, with the same ports and credentials setting after a clone, or the Agent Client Collectors will not be able to communicate with the MID Server and instance any more.

PRB1408738 should have fixed this by the GA release.

Full Clones

All the settings related to Clone Exclude and Preserve settings are ignored if the "Exclude tables specified in Exclusion List" and "Exclude audit and log data" options are checked when requesting the clone (PRB1251951). We call doing that a Full Clone, and you can expect to have to set up all MID Server related things again after the clone.

Article Information

Last Updated:2020-07-02 04:35:03
Published:2020-07-02