Notifications

147 views

Description

Occasionally, there can be an issue with a MID Server installation causing the MID Server to lose communication with the instance and go Down. In this situation, the Restart, Grab Logs, Get MID Thread Dump functions and other useful tricks for debugging a MID Server can't be performed remotely via the instance.

In some of those situations, the customer may not immediately have access to the host server in order to help with the debugging, such as checking the agent logs and attaching them to a support incident for ServiceNow to help. 

Here are some ideas that may avoid delays in starting the investigation for high impact incidents. 

Note: There is the opportunity to screw up badly when running from the command line on a customer's host, so get agreement and document everything. These are all publicly documented features, not hidden back-doors, but customers may not be aware we have these in our toolbox.

Procedure

Use another 'Up' MID Server running on the same host

Often more than one MID Server is installed on the same host. Perhaps a MID Server for Discovery and a separate one for LDAP, or another for a sub-production instance. If one of those is Up, then access to files and services is possible.

A MID server service usually logs in as a domain account with local administrator privileges (unless the log in as user has been changed, or the post-Paris MSI installer was used). Anything, including reading configuration and log files, and restarting windows services, is possible if "Command" jobs are sent. "Command" topic is a documented feature.

In general the "Command" jobs are set up like this:

  1. Open a new blank ECC Queue record form - /ecc_queue.do
  2. Fill in the fields like so:
    • Agent = mid.server.<MID Server name>
      remembering to use the MID Server that is still Up
    • Topic = Command
    • Name = <your command>
    • Queue = Output
    • State = Ready
    • Sequence = (clear this value)
  3. Submit
  4. Look in the ECC Queue table for the Input response from that output. The output from the commands will be in the Payload.
    /ecc_queue_list.do?sysparm_query=topic%3dCommand

Useful commands to use in the Name field:

dir /s /b \mid.jar
dir /s /b \wrapper-override.conf
dir /s /b \config.xml

Find all MID Server installations on the same disk. Only a MID Server installation will have mid.jar. 
Other java applications may also have config.xml and wrapper-override.conf files, so this may bring back other things.

type <agent path>\logs\<log filename>
type C:\MID Servers\Prod_Disco_MID\agent\config.xml
type C:\MID Servers\Prod_Disco_MID\agent\conf\wrapper-override.conf

Once you know what you have, you can list the contents of the settings files to figure out which MID Server is which.
config.xml will have a URL and Name parameter for the instance and mid server name.
wrapper-override.conf will have the windows service wrapper.name and wrapper.displayname.

type C:\MID Servers\Prod_Disco_MID\agent\logs\agent0.log.0
type C:\MID Servers\Prod_Disco_MID\agent\logs\wrapper.log

List log files. The agent log is the detailed MID Server applications log, including the AutoUpgrade logs. The Wrapper log has details of application start/stop, Upgrade Logs, and exceptions.

net stop "<wrapper.name>"
net stop "snc_mid_Prod Disco MID"

Stop a MID Server Windows Service. 
net start "<wrapper.name>"Start a MID Server Windows Service
tasklist -vList all running processes, with their executable name, PID, memory usage, and run as user
wmic service
List all running and no-running services, wmic service gives all parameters of the service including PID, folders, display/service names, status. Finds MID Servers, WMI Collectors, Upgrade services, and helps identify the Anti-Virus running on the host.

It is possible to paste multiple lines into the Name field. It won't look right on the form, but the new line characters do get sent, and the lines will be run one after each other on the target.

Use the Debug Mode Command Prompt feature of the Service Mapping Pattern Designer via another 'Up' MID Server

This is only available if Discovery or Service Mapping is installed, and if a Discovery Credential is available for the MID Server Host. You can explore both remote hosts and MID Servers.

This 'Command Prompt' feature can be useful for seeing what's going on with a Down MID Server, assuming there is a credential available for accessing the MID Server host. 

Command Prompt and Debug Mode is a documented feature, and similar commands to the above could be used with it.

You can go directly to the Command Prompt page from: /SaCmdManager.do?ip=<ip address of mid server host>

Release or Environment

Any for Command topic. Since approx Geneva for SaCmdManager.

Article Information

Last Updated:2020-09-22 05:57:19
Published:2020-09-22