Tricks for Remote Debugging of a "Down" MID Server via the instance

Issue

Introduction
Procedure

Introduction

Occasionally, there can be an issue with a MID Server installation causing the MID Server to lose communication with the instance and go Down. In this situation, the Restart, Grab Logs, Get MID Thread Dump functions and other useful tricks for debugging a MID Server can't be performed remotely via the instance.

In some of those situations, the customer may not immediately have access to the host server in order to help with the debugging, such as checking the agent logs and attaching them to a support incident for ServiceNow to help.

Here are some ideas that may avoid delays in starting the investigation for high impact incidents.

Note: There is the opportunity to screw up badly when running from the command line on a customer's host, so get agreement and document everything. These are all publicly documented features, not hidden back-doors, but customers may not be aware we have these in our toolbox.

Procedure

Use another 'Up' MID Server running on the same host

Often more than one MID Server is installed on the same host. Perhaps a MID Server for Discovery and a separate one for LDAP, or another for a sub-production instance. If one of those is Up, then access to files and services is possible.

A MID server service usually logs in as a domain account with local administrator privileges (unless the log in as user has been changed, or the post-Paris MSI installer was used). Anything, including reading configuration and log files, and restarting windows services, is possible if "Command" jobs are sent. "Command" topic is a documented feature.

In general the "Command" jobs are set up like this:

Open a new blank ECC Queue record form - /ecc_queue.do
Fill in the fields like so:
- Agent = mid.server.<MID Server name>
  remembering to use the MID Server that is still Up
- Topic = Command
- Name = <your command>
- Queue = Output
- State = Ready
- Sequence = (clear this value)
Submit
Look in the ECC Queue table for the Input response from that output. The output from the commands will be in the Payload.
/ecc_queue_list.do?sysparm_query=topic%3dCommand

Useful commands to use in the Name field:

dir /s /b \mid.jar dir /s /b \wrapper-override.conf dir /s /b \config.xml	Find all MID Server installations on the same disk. Only a MID Server installation will have mid.jar. Other java applications may also have config.xml and wrapper-override.conf files, so this may bring back other things.
type <agent path>\logs\<log filename> type C:\MID Servers\Prod_Disco_MID\agent\config.xml type C:\MID Servers\Prod_Disco_MID\agent\conf\wrapper-override.conf	Once you know what you have, you can list the contents of the settings files to figure out which MID Server is which. config.xml will have a URL and Name parameter for the instance and mid server name. wrapper-override.conf will have the windows service wrapper.name and wrapper.displayname.
type C:\MID Servers\Prod_Disco_MID\agent\logs\agent0.log.0 type C:\MID Servers\Prod_Disco_MID\agent\logs\wrapper.log	List log files. The agent log is the detailed MID Server applications log, including the AutoUpgrade logs. The Wrapper log has details of application start/stop, Upgrade Logs, and exceptions.
net stop "<wrapper.name>" net stop "snc_mid_Prod Disco MID"	Stop a MID Server Windows Service.
net start "<wrapper.name>"	Start a MID Server Windows Service
tasklist -v	List all running processes, with their executable name, PID, memory usage, and run as user
wmic service	List all running and no-running services, wmic service gives all parameters of the service including PID, folders, display/service names, status. Finds MID Servers, WMI Collectors, Upgrade services, and helps identify the Anti-Virus running on the host.

It is possible to paste multiple lines into the Name field. It won't look right on the form, but the new line characters do get sent, and the lines will be run one after each other on the target.

Use the Debug Mode Command Prompt feature of the Service Mapping Pattern Designer via another 'Up' MID Server

This is only available if Discovery or Service Mapping is installed, and if a Discovery Credential is available for the MID Server Host. You can explore both remote hosts and MID Servers.

This 'Command Prompt' feature can be useful for seeing what's going on with a Down MID Server, assuming there is a credential available for accessing the MID Server host.

Command Prompt and Debug Mode is a documented feature, also mentioned in KB0725806 CLI console for service mapping using "SaCmdManager", and similar commands to the above could be used with it.

You can go directly to the Command Prompt page from: /SaCmdManager.do?ip=<ip address of mid server host>

There is a big limitation with this tool, which is that you cannot specify which MID Server should be used to run the probe. So use this instead....

Use the Command Validation Tool from the Pattern Designer Enhancements app

Store link: https://store.servicenow.com/sn_appstore_store.do#!/store/application/f73627bf89228510f877b828becf35e9/
Release note: https://docs.servicenow.com/bundle/store-release-notes/page/release-notes/store/it-operations-management/store-rn-itom-pattern-designer-enhancements.html

Documentation is in this KB article: KB1123625 Command Validation Tool

This was released in 2023 and is compatible with Tokyo, San Diego Patch 5, Rome Patch 8, and later, and can be installed if Discovery, Service Mapping, or Cloud Management are installed.

Release

Any for Command topic. Since approx Geneva for SaCmdManager. Since October 2023 for