Credentials troubleshooting on Discovery, Service Mapping, Orchestration.

Overview

Throughout this article there are troubleshooting steps as well as links for more in depth articles for specific credentials types. 

Credentials

In order to discover a device or perform orchestration activities, add credentials to the credentials table. These credential records specify a username, a password, a kind of credential (Windows, SSH, ...), and MID Servers that are allowed to use this credential.

When the MID Server starts or when a credential is modified, the MID Server downloads and caches all available credentials.

Credentials for Discovery, Service Mapping, and Orchestration all point to the same credentials table.

General Troubleshooting

Test the credential:

1. Navigate to the credentials table
2. Select the credential
3. Click on Test credential
4. Populate the form fields for the credential test
5. Click OK

If the credential test fails, check that the credential has the correct username and password populated; a typographical error is very often the cause for the credential error. Once the credential fields are confirmed correct, if the credential error continues debugging can be turned on in the MID server for further investigation. The MID server logs should then be reviewed after the issue is reproduced with debug turned on. However, before turning debug confirm the MID server can communicate to the target as seen on the next steps.

Ping the device:

A ping confirms the device is available in the network.

• Log into the MID server host, open up the command prompt and run:

ping <ip_address>

Please consult the target device administrator if ping is not successful.

Obs: Some environments may have ICMP requests disabled which could cause ping to fail.

Telnet into the port used for the credential test:

If a ping is successful, then the device is available in the network and reachable from the MID server host. However protocols used to communicate to the target hosts must connect to a specified port. Such ports may not be open on the target host or on the network path used to reach the device. A telnet test will confirm the whether the port is open or not.

telnet <ip_address> <port_number>

The following are some of the ports used out of box, however a system administrator could change these ports.

WMI: 135
SSH: 22
VCenter: 443
WinRM: 5985
WBEM: 5989
LDAP: 389

If the telnet test fails, an error message would be displayed, please consult the target device administrator and the network administrator to confirm that:

1. The firewall on the device allows traffic on the port tested by telnet
2. There are no network firewalls blocking traffic from the MID server to the target host on the port tested

Note: Telnet is an application that operates using the TCP protocol. UDP connectivity can not be tested using Telnet.

Collecting MID Server Debug logs

To turn on debugging on the MID server:

1. Navigate to the MID server list, MID Server > Servers
2. Select the MID server used for the failed discovery or orchestration activity
3. Select the Configuration Parameters related list and click New
4. Select Parameter Name = mid.log.level and set the value to debug. Once debugging is complete, set this value back to info

To collect the MID server logs:

1. On the MID server record click on Grab MID logs
2. Click on the input records displayed

Windows Credential

A simple Powershell WMI query directly from the MID Server to the remote machine can be used to test access and permissions. 

Open a PowerShell command line on the host where the MID server is being used and run the following:

gwmi win32_operatingsystem -computer 192.168.200.14 -credential 'LOCALDOMAIN\mid'

Substitute LOCALDOMAIN\mid by the credential to test, and 192.168.200.14 with the target IP address. The expected result is something similar to:

SystemDirectory : C:\Windows\system32
Organization    :
BuildNumber     : 6001
RegisteredUser  : Windows User
SerialNumber    : 12345-OEM-1234567-12345
Version         : 6.0.6001

If the WMI command above fails, either the credential is incorrect or lacks permission. We advise your windows admin team to further investigate the issue. If basic WMI queries from the MID server to the target hosts fails, then discovery and orchestration activities would not be successful.

Further Documentation

Troubleshooting articles for windows credential errors:

KB0549834 - MID Server: troubleshooting WMI/Powershell issues - Credentials

KB0549830 - Windows Discovery - Troubleshooting WMI/Powershell issues on the remote machine

KB0549828 - WMI, PowerShell, and Windows Firewalls

Permission requirements for windows credentials:

Permission requirements for Windows credentials

Discovery Windows probes and permissions

Additional Discovery probe permissions

SSH Credential

Confirm Authentication:

To troubleshoot SSH credential errors, use any SSH client and connect to the target IP address from the MID server being used to ensure the account can successfully login to the target host, putty, for example. Make sure to use the same username/password combination as set in the credentials table. Access to the target system is necessary for any discovery or orchestration activity to work. 

Confirm Authorization/Permission:

A user may be able to login to a target system however may not have permission to run the command attempted by either discovery or orchestration. See the following links and confirm whether the user meets the requirments.

Access Requirements for Non-Root Credentials

UNIX and Linux commands requiring root privileges for Discovery and Orchestration

To check what commands a user can run on a Unix based device, log into the target host with the user and run the following command:

sudo -l

The output shows what priviledged commands a user can run using sudo.

Check what implementation of SSH is being used:

mid.ssh.use_snc enables the ServiceNow SSH client (SNCSSH) on individual MID Servers. SNCSSH is a ServiceNow implementation of an SSH client and is active by default for all MID Servers on new instances, via a MID server property. Enabling the ServiceNow SSH client disables the legacy J2SSH client.

Private Key Credentials: 

disco ALL=(root) NOPASSWD:/usr/sbin/dmidecode,/usr/sbin/lsof,/sbin/ifconfig.

Finally: 

SNMP Credential

SNMP uses UDP, which does not create a virtual connection to the target host as TCP, and no reply may be given if a credential is not correct or authorized depending on the verson used. A timeout can be seen in the discovery log when an invalid credential is used.

A couple of reasons for a timeout are:

1. The credential is not valid
2. The network or target server is too busy and does not return the OIDs within the timeout

Run an SNMP query to the target:

Using an SNMP tool, from the MID server, query OID 1.3.6.1.2.1.1.1. This OID is the sysDescr and will return a description of the device. 

The following example uses SnmpWalk.exe, however has publi is an incorrect community string, the correct public string for this example should be public

C:\SNMPWalk>.\SnmpWalk.exe -r:10.127.212.181 -c:"publi" -os:.1.3.6.1.2.1.1 -op:.1.3.6.1.2.1.1.1.0

%Failed to get value of SNMP variable. Timedout.

As seen above there is no credential failure error. Instead of an error the query eventually times out.

In the following example the public string was corrected, public

C:\SNMPWalk>.\SnmpWalk.exe -r:10.127.212.181 -c:"public" -os:.1.3.6.1.2.1.1 -op:.1.3.6.1.2.1.1.1.0

OID=.1.3.6.1.2.1.1.1.0, Type=OctetString, Value=Linux Linux-Tomcat 3.10.0-327.el7.x86_64 31 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64

As seen above, once the public string was corrected then the sysDescr was returned, only part of it is shown above.

SNMP Parameters:

The "timeout" for an SNMP request can be increased via the "timeout" probe parameter. The default value is 1500, 1.5s. Try increasing the timeout if a credential is known to be correct and still no OIDs are returned. For tabular data try the "use_getbulk" parameter to improve efficiency.

See more information for SNMP probe parameters on SNMP probes.

No result returned from probe:

In some cases a warning can be seen in the discovery log stating that No result returned from probe. This is seen when an input returns empty for the OID queried by the probe. This happens because a device does not have any results to return for the specified OID, and if that is the case this is expected. If there is suspicion that discovery may be wrong, a query for the same oid against the target IP address can be run to check on the output. 

VMWare Credential 

Confirm Authentication:

Confirm the same account configured in the credentials table can log into the VCenter target:

1. Log into the MID server host
2. Open up a browser and navigate to https://<V-Center_IP_Address>/mob, replace the address with the IP address of the VCenter server
3. Make sure to use the same exact username/password combination and the same format as seen in the credentials table record

If the test above fails have your vmware team further troubleshoot or provide access to the credential. If the test above is successful, however discovery still fails, then follow the steps for Collecting MID server debug logs for further investigation.

Further Documentation