Issue
Symptoms
-Discovery schedules stuck or getting cancelled when using external credentials.
Release
All Releases
Cause
-If you have too many credential ID's in your credentials table, all the credential ID's will be tested for your discovery. There is only one credential resolver proxy that will test these credentials against Cyberark. Having too many credential ID's associated with all the mid servers will lock this thread and eventually cause delay in processing the schedule.
-There may be connectivity issues between Mid server to cyberArk that may cause delay in reading the data from CyberArk.
-Sample thread Dump
"Worker-Standard:HorizontalDiscoveryProbe" #137 daemon prio=5 os_prio=0 tid=0x0000000022595800 nid=0x7a4 runnable [0x0000000027c3e000]
2019/01/02 12:58:49 | java.lang.Thread.State: RUNNABLE
2019/01/02 12:58:49 | at java.net.SocketInputStream.socketRead0(Native Method)
2019/01/02 12:58:49 | at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2019/01/02 12:58:49 | at java.net.SocketInputStream.read(SocketInputStream.java:171)
2019/01/02 12:58:49 | at java.net.SocketInputStream.read(SocketInputStream.java:141)
2019/01/02 12:58:49 | at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
2019/01/02 12:58:49 | at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
2019/01/02 12:58:49 | at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
2019/01/02 12:58:49 | - locked <0x0000000771ceaef8> (a java.io.InputStreamReader)
2019/01/02 12:58:49 | at java.io.InputStreamReader.read(InputStreamReader.java:184)
2019/01/02 12:58:49 | at java.io.BufferedReader.fill(BufferedReader.java:161)
2019/01/02 12:58:49 | at java.io.BufferedReader.read1(BufferedReader.java:212)
2019/01/02 12:58:49 | at java.io.BufferedReader.read(BufferedReader.java:286)
2019/01/02 12:58:49 | - locked <0x0000000771ceaef8> (a java.io.InputStreamReader)
2019/01/02 12:58:49 | at java.io.Reader.read(Reader.java:140)
2019/01/02 12:58:49 | at d.c.b(c.java:71)
2019/01/02 12:58:49 | at k.l.c(l.java:4)
2019/01/02 12:58:49 | at k.g.a(g.java:107)
2019/01/02 12:58:49 | at k.i.a(i.java:53)
2019/01/02 12:58:49 | at javapasswordsdk.PasswordSDK.getPassword(PasswordSDK.java:1)
2019/01/02 12:58:49 | at com.service_now.mid.services.credential.cyberark.CyberArkPasswordAccess.getPassword(CyberArkPasswordAccess.java:26)
2019/01/02 12:58:49 | at com.service_now.mid.services.credential.cyberark.CyberArkAccess.getCred(CyberArkAccess.java:216)
2019/01/02 12:58:49 | at com.service_now.mid.services.credential.cyberark.CredentialResolver.resolve(CredentialResolver.java:88)
2019/01/02 12:58:49 | at com.service_now.mid.services.CredentialResolverProxy.resolve(CredentialResolverProxy.java:227)
2019/01/02 12:58:49 | - locked <0x00000006c386b8e0> (a com.service_now.mid.services.CredentialResolverProxy)
2019/01/02 12:58:49 | at com.service_now.mid.creds.provider.standard.HighSecurityCredential.callResolver(HighSecurityCredential.java:43)
2019/01/02 12:58:49 | at com.service_now.mid.creds.provider.standard.HighSecurityCredential.<init>(HighSecurityCredential.java:33)
2019/01/02 12:58:49 | at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.iterator(StandardCredentialsProvider.java:153)
2019/01/02 12:58:49 | - locked <0x00000006c0428f08> (a com.service_now.mid.creds.provider.standard.StandardCredentialsProvider)
2019/01/02 12:58:49 | at com.snc.sw.providers.DiscoveryProviderFactory.getApplicativeCredentials(DiscoveryProviderFactory.java:105)
2019/01/02 12:58:49 | at com.snc.sw.context.ExecutionContextFactory.create(ExecutionContextFactory.java:79)
2019/01/02 12:58:49 | at com.snc.sw.context.ExecutionContextFactory.createAppDiscovery(ExecutionContextFactory.java:129)
2019/01/02 12:58:49 | at com.snc.sw.context.ExecutionContextFactory.createAppDiscovery(ExecutionContextFactory.java:137)
2019/01/02 12:58:49 | at com.snc.sw.context.ExecutionContextFactory.createAppDiscovery(ExecutionContextFactory.java:143)
2019/01/02 12:58:49 | at com.service_now.mid.probe.HorizontalDiscoveryProbe.runProbe(HorizontalDiscoveryProbe.java:81)
2019/01/02 12:58:49 | at com.service_now.mid.probe.ServiceWatchProbe.probe(ServiceWatchProbe.java:35)
2019/01/02 12:58:49 | at com.service_now.mid.probe.AProbe.process(AProbe.java:84)
2019/01/02 12:58:49 | at com.service_now.mid.queue_worker.AWorker.runWorker(AWorker.java:125)
2019/01/02 12:58:49 | at com.service_now.mid.queue_worker.AWorkerThread.run(AWorkerThread.java:20)
2019/01/02 12:58:49 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019/01/02 12:58:49 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
Resolution
-Always make sure the credentials ID's are configured properly with respective mid servers.
-Check to see if there is in delay in reading the data from CyberArk or if the thread is stuck reading the data.
