Issue
This article provides info about specific credential types and troubleshooting steps for common credentials and permissions issues for Discovery, Orchestration, and Service Mapping.
For a walkthrough on troubleshooting credentials issues with Discovery, view the video below:
Release
All
Resolution
Credentials
In order to discover a device or perform orchestration activities, add credentials to the credentials table. These credential records specify a username, a password, a kind of credential (Windows, SSH, ...), and MID Servers that are allowed to use this credential.
When the MID Server starts or when a credential is modified, the MID Server downloads and caches all available credentials.
Credentials for Discovery, Service Mapping, and Orchestration all point to the same credentials table.
General Troubleshooting
General troubleshooting will involve three main actions: Test the credential, ping the device, and Telnet into the port used for the credential test. See each section below for information about each action and steps to perform them correctly.
Test the credential
- Navigate to the credentials table.
- Select the credential.
- Click on Test credential.
- Populate the form fields for the credential test.
- Click OK.
If the credential test fails, check that the credential has the correct username and password populated; a typographical error is very often the cause for the credential error. Once the credential fields are confirmed correct, if the credential error continues debugging can be turned on in the MID server for further investigation. The MID server logs should then be reviewed after the issue is reproduced with debug turned on. However, before turning debug confirm the MID server can communicate to the target as seen on the next steps.
Ping the device
A ping confirms the device is available in the network.
- Log into the MID server host, open up the command prompt and run:
ping <ip_address>
Please consult the target device administrator if ping is not successful.
Obs: Some environments may have ICMP requests disabled which could cause ping to fail.
Telnet into the port used for the credential test
If a ping is successful, then the device is available in the network and reachable from the MID server host. However protocols used to communicate to the target hosts must connect to a specified port. Such ports may not be open on the target host or on the network path used to reach the device. A telnet test will confirm the whether the port is open or not.
telnet <ip_address> <port_number>
The following are some of the ports used out of box, however a system administrator could change these ports.
WMI: 135
SSH: 22
VCenter: 443
WinRM: 5985
WBEM: 5989
LDAP: 389
If the telnet test fails, an error message would be displayed, please consult the target device administrator and the network administrator to confirm that:
- The firewall on the device allows traffic on the port tested by telnet
- There are no network firewalls blocking traffic from the MID server to the target host on the port tested
Note: Telnet is an application that operates using the TCP protocol. UDP connectivity can not be tested using Telnet.
Collecting MID Server Debug logs
To turn on debugging on the MID server, follow these steps:
- Navigate to the MID server list, MID Server > Servers.
- Select the MID server used for the failed discovery or orchestration activity.
- Select the Configuration Parameters related list and click New.
- Select Parameter Name = mid.log.level and set the value to debug. Once debugging is complete, set this value back to info.
To collect the MID server logs, follow these steps:
- On the MID server record click on Grab MID logs.
- Click on the input records displayed.
Windows Credentials
A simple Powershell WMI query directly from the MID Server to the remote machine can be used to test access and permissions.
Open a PowerShell command line on the host where the MID server is being used and run the following:
gwmi win32_operatingsystem -computer 192.168.200.14 -credential 'LOCALDOMAIN\mid'
Substitute LOCALDOMAIN\mid by the credential to test, and 192.168.200.14 with the target IP address. The expected result would be similar to:
SystemDirectory : C:\Windows\system32
Organization :
BuildNumber : 6001
RegisteredUser : Windows User
SerialNumber : 12345-OEM-1234567-12345
Version : 6.0.6001
If the WMI command above fails, either the credential is incorrect or lacks permission. We advise your windows admin team to further investigate the issue. If basic WMI queries from the MID server to the target hosts fails, then discovery and orchestration activities would not be successful.
Further Documentation on ServiceNow Windows Credentials
Below is a collection of in-depth troubleshooting articles for windows credential errors.
- KB0549834 - MID Server: troubleshooting WMI/Powershell issues - Credentials
- KB0549830 - Windows Discovery - Troubleshooting WMI/Powershell issues on the remote machine
- KB0549828 - WMI, PowerShell, and Windows Firewalls
Below is a collection of product documentation articles containing detailed info about permission requirements for windows credentials.
- Permission requirements for Windows credentials
- Discovery Windows probes and permissions
- Additional Discovery probe permissions
SSH Credential
Confirm Authentication
To troubleshoot SSH credential errors, use any SSH client and connect to the target IP address from the MID server being used to ensure the account can successfully login to the target host, putty, for example. Make sure to use the same username/password combination as set in the credentials table. Access to the target system is necessary for any discovery or orchestration activity to work.
Confirm Authorization/Permission
A user may be able to login to a target system however may not have permission to run the command attempted by either discovery or orchestration. See the following links and confirm whether the user meets the requirements.
- Access Requirements for Non-Root Credentials
- UNIX and Linux commands requiring root privileges for Discovery and Orchestration
To check what commands a user can run on a Unix based device, log into the target host with the user and run the following command:
sudo -l
The output shows what privileged commands a user can run using sudo.
Check what implementation of SSH is being used
mid.ssh.use_snc enables the ServiceNow SSH client (SNCSSH) on individual MID Servers. SNCSSH is a ServiceNow implementation of an SSH client and is active by default for all MID Servers on new instances, via a mid.ssh.use_snc. Enabling the ServiceNow SSH client disables the legacy J2SSH client.
Private Key Credentials
disco ALL=(root) NOPASSWD:/usr/sbin/dmidecode,/usr/sbin/lsof,/sbin/ifconfig.
In the example above, the user disco can run dmidecode via sudo without providing a password. If a command fails when using a private key credential, confirm the user can run the command without providing a password.
SNMP Credential
SNMP uses UDP, which does not create a virtual connection to the target host as TCP, and no reply may be given if a credential is not correct or authorized depending on the version used. A timeout can be seen in the discovery log when an invalid credential is used.
A couple of reasons for a timeout are:
- The credential is not valid
- The network or target server is too busy and does not return the OIDs within the timeout
Run an SNMP query to the target
Using an SNMP tool, from the MID server, query OID 1.3.6.1.2.1.1.1. This OID is the sysDescr and will return a description of the device.
The following example uses SnmpWalk.exe, however has publi is an incorrect community string, the correct public string for this example should be public.
C:\SNMPWalk>.\SnmpWalk.exe -r:10.127.212.181 -c:"publi" -os:.1.3.6.1.2.1.1 -op:.1.3.6.1.2.1.1.1.0
%Failed to get value of SNMP variable. Timedout.
As seen above there is no credential failure error. Instead of an error the query eventually times out.
In the following example the public string was corrected, public.
C:\SNMPWalk>.\SnmpWalk.exe -r:10.127.212.181 -c:"public" -os:.1.3.6.1.2.1.1 -op:.1.3.6.1.2.1.1.1.0
OID=.1.3.6.1.2.1.1.1.0, Type=OctetString, Value=Linux Linux-Tomcat 3.10.0-327.el7.x86_64 31 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64
As seen above, once the public string was corrected then the sysDescr was returned, only part of it is shown above.
SNMP Parameters
The "timeout" for an SNMP request can be increased via the "timeout" probe parameter. The default value is 1500, 1.5s. Try increasing the timeout if a credential is known to be correct and still no OIDs are returned. For tabular data try the "use_getbulk" parameter to improve efficiency.
See more information for SNMP probe parameters on SNMP probes.
No result returned from probe
In some cases a warning can be seen in the discovery log stating No result returned from probe. This is seen when an input returns empty for the OID queried by the probe. This happens because a device does not have any results to return for the specified OID, and if that is the case this is expected. If there is suspicion that discovery may be wrong, a query for the same oid against the target IP address can be run to check on the output.
Further Documentation regarding SNMP credentials
VMWare Credential
Confirm Authentication
Confirm the same account configured in the credentials table can log into the VCenter target:
- Log into the MID server host
- Open up a browser and navigate to https://<V-Center_IP_Address>/mob, replace the address with the IP address of the VCenter server
- Make sure to use the same exact username/password combination and the same format as seen in the credentials table record
If the test above fails have your vmware team further troubleshoot or provide access to the credential. If the test above is successful, however discovery still fails, then follow the steps for Collecting MID server debug logs for further investigation.