Notifications

76 views

Description

Symptoms

Using the Madrid release or above, an Edge Proxy is brought online and the "Authentication" is set to "Unauthenticated".

The proxy is selected and you select the "Authenticate" button, but the proxy does not move from "Unauthenticated" to "Authenticated".

Checking the Edge Proxy <installation directory>/logs/edgeencryption.log shows this during the Authentication attempt:

2019-07-12 07:44:34,298 WARN Received an error response code Signed certificate in response to 500 from the ServiceNow instance : "instance1.service-now.com". Message: java.security.KeyStoreException: Uninitialized keystore | Detail:

Checking the instance node logs for the same time will show messages like this:

2019-07-12 04:44:32 (253) http-32 New transaction 0CF7839ADB6A3740D5CFF2131F9619C9 #21221 /api/sn_edge_encryption/v1/sync/authenticate
2019-07-12 04:44:32 (271) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a #21221 /api/sn_edge_encryption/v1/sync/authenticate Parameters -------------------------
edge_encryption_pool=
api=api
message=O774turqrWPROsh7Lk3Z15jKvrjw5WTs0JlQhSe3KHFd0qcddaF1W8s2NN1R098dkOdBDP21yrpWnZ8z9D8QrGyHlNKT5Rcx7iCt4WJoy94TLTZRflNSZ0qg4Vc3ZVPitEYdcckDaK9BhSyLbALbJC2idDm4afmKlCE-Vct79Wsnllw9HDT7eYUXMxN5If5505Zay6o6ygoSlA6osmEmnej7nw1wCH2MQrmcHTItkrXxJfrLZCEIhQD6OQrSc_c5Ut5A4SB9iSClx_FqZVCBtXuPqdr2CcDgPNwAcy6CIoKoeMSkWKUNPb0DLK44nxNvRsOTlB-7mhpWLppTOoic-20lCO7QRxu8FYcU5ZlideUqiY7ki2hekw7HEnTkWyRkvwbUbIfRW69hXChpmHYj0xsUlfXBDVlWiGyCo_fCO8emdy4aRvPh0A6CttNIkleB1Rixz4O9LcPfd-jbenowgJ4YKoR3U2oKxl_vV_DJziyj5wtuctcLFpkrWYTnZl_b
hash=
2019-07-12 04:44:32 (271) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a *** Start #21221 /api/sn_edge_encryption/v1/sync/authenticate, user: edge
2019-07-12 04:44:32 (348) http-39 New transaction 3267039ADB6A3740D5CFF2131F9619C7 #21222 send /amb/meta/disconnect
2019-07-12 04:44:32 (394) AMB_SEND-thread-36 3267039ADB6A3740D5CFF2131F9619C7 txid=2d08879adb6a *** Start #21222 send /amb/meta/disconnect, user: a.b@snc
2019-07-12 04:44:32 (394) AMB_SEND-thread-36 3267039ADB6A3740D5CFF2131F9619C7 txid=2d08879adb6a [AMB] AMBSessionManager cometd_session_id=5315xv7xd1qvzwd1ci3ozv0pzlxm Session removed
2019-07-12 04:44:32 (394) AMB_SEND-thread-36 3267039ADB6A3740D5CFF2131F9619C7 txid=2d08879adb6a *** End #21222 send /amb/meta/disconnect, user: a.b@snc, total time: 0:00:00.046, processing time: 0:00:00.046, SQL time: 0:00:00.001 (count: 1), source: 199.91.140.61
2019-07-12 04:44:32 (411) http-36 New transaction 3267039ADB6A3740D5CFF2131F9619C7 #21223 /api/now/ui/date_time/legacy
2019-07-12 04:44:32 (422) Default-thread-206 SYSTEM txid=35084b9adb6a WARNING *** WARNING *** Cookie token not in database: SCv3:l9JfJYLGUJ9wq8VExfxycslGKCGsvhQx:ysQo0cuHrP3x8dj3YJFcXChcqW1NePzt5+D5JOPy3yU=
2019-07-12 04:44:32 (422) Default-thread-206 SYSTEM txid=35084b9adb6a WARNING *** WARNING *** aborted activity cookie update: U0N2MzpsOUpmSllMR1VKOXdxOFZFeGZ4eWNzbEdLQ0dzdmhReDp5c1FvMGN1SHJQM3g4ZGozWUpGY1hDaGNxVzFOZVB6dDUrRDVKT1B5M3lVPQ==
2019-07-12 04:44:32 (423) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a Bypassing ACL checks for a public page: /api/now/ui/date_time/legacy
2019-07-12 04:44:32 (423) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a #21223 /api/now/ui/date_time/legacy Parameters -------------------------
api=api
2019-07-12 04:44:32 (423) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a *** Start #21223 /api/now/ui/date_time/legacy, user: a.b@snc
2019-07-12 04:44:32 (427) Default-thread-206 3267039ADB6A3740D5CFF2131F9619C7 txid=35084b9adb6a *** End #21223 /api/now/ui/date_time/legacy, user: a.b@snc, total time: 0:00:00.014, processing time: 0:00:00.014, SQL time: 0:00:00.001 (count: 4), source: 199.91.140.61 , type:rest, method:GET, api_name:now/ui, resource:now/ui/date_time/legacy, version:Default, user_id:a.b@snc, response_status:200
2019-07-12 04:44:32 (523) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** string may not be encrypted : Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2019-07-12 04:44:32 (524) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a SEVERE *** ERROR *** Attachment is missing for certificate sn_edge_instance_selfcert - 8d80f1a6dbcc3b409063f4eabf961922
2019-07-12 04:44:32 (528) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** Evaluator: java.security.KeyStoreException: Uninitialized keystore
Caused by error in sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script at line 22
java.security.KeyStore.getKey(KeyStore.java:1021)
com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.fetchSigningKey(AuthenticationProcessorAPI.java:393)
com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.jsStaticFunction_generateSignedCertificate(AuthenticationProcessorAPI.java:184)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670)
org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582)
org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_process_1(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:22)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_script_0(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:1)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563)
org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3429)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.exec(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201)
com.glide.rest.service.custom.CustomService.runScript(CustomService.java:96)
com.glide.rest.service.custom.CustomService.execute(CustomService.java:83)
com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:37)
com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:290)
com.glide.processors.AProcessor.runProcessor(AProcessor.java:531)
com.glide.processors.AProcessor.processTransaction(AProcessor.java:229)
com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:188)
com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:177)
com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31)
com.glide.sys.Transaction.run(Transaction.java:2147)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2019-07-12 04:44:32 (528) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** Evaluator: java.security.KeyStoreException: Uninitialized keystore
Caused by error in sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script at line 1
java.security.KeyStore.getKey(KeyStore.java:1021)
com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.fetchSigningKey(AuthenticationProcessorAPI.java:393)
com.glide.edgeencryption.authentication.AuthenticationProcessorAPI.jsStaticFunction_generateSignedCertificate(AuthenticationProcessorAPI.java:184)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670)
org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582)
org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_process_1(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:22)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_script_0(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:1)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563)
org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3429)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.exec(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201)
com.glide.rest.service.custom.CustomService.runScript(CustomService.java:96)
com.glide.rest.service.custom.CustomService.execute(CustomService.java:83)
com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:37)
com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:290)
com.glide.processors.AProcessor.runProcessor(AProcessor.java:531)
com.glide.processors.AProcessor.processTransaction(AProcessor.java:229)
com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:188)
com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:177)
com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31)
com.glide.sys.Transaction.run(Transaction.java:2147)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

2019-07-12 04:44:32 (530) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a SEVERE *** ERROR *** java.security.KeyStoreException: Uninitialized keystore
com.glide.rest.domain.ServiceException: java.security.KeyStoreException: Uninitialized keystore
at com.glide.rest.service.custom.CustomServiceExceptionResolver.throwServiceException(CustomServiceExceptionResolver.java:82)
at com.glide.rest.service.custom.CustomServiceExceptionResolver.throwServiceException(CustomServiceExceptionResolver.java:77)
at com.glide.rest.service.custom.CustomServiceExceptionResolver.resolveForException(CustomServiceExceptionResolver.java:39)
at com.glide.rest.service.custom.CustomServiceResultHandler.handle(CustomServiceResultHandler.java:22)
at com.glide.rest.service.custom.CustomService.execute(CustomService.java:84)
at com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:37)
at com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:290)
at com.glide.processors.AProcessor.runProcessor(AProcessor.java:531)
at com.glide.processors.AProcessor.processTransaction(AProcessor.java:229)
at com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:188)
at com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:177)
at com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31)
at com.glide.sys.Transaction.run(Transaction.java:2147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.mozilla.javascript.JavaScriptException: java.security.KeyStoreException: Uninitialized keystore
at org.mozilla.javascript.Context.makeJavaScriptException(Context.java:1935)
at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Context.java:1921)
at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:143)
at org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670)
at org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614)
at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582)
at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_process_1(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:22)
at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
at org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2651)
at org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411._c_script_0(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script:1)
at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3429)
at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.call(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
at org.mozilla.javascript.gen.sys_ws_operation_bf46ad5e53402300cd1eddeeff7b12aa_operation_script_2411.exec(sys_ws_operation.bf46ad5e53402300cd1eddeeff7b12aa.operation_script)
at com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279)
at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118)
at com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82)
at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309)
at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214)
at com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201)
at com.glide.rest.service.custom.CustomService.runScript(CustomService.java:96)
at com.glide.rest.service.custom.CustomService.execute(CustomService.java:83)
... 11 more

2019-07-12 04:44:32 (531) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a WARNING *** WARNING *** #21221 [REST API] RESTAPIProcessor : Handling exception java.security.KeyStoreException: Uninitialized keystore
2019-07-12 04:44:32 (532) Edge Encryption-thread-1 0CF7839ADB6A3740D5CFF2131F9619C9 txid=2908879adb6a *** End #21221 /api/sn_edge_encryption/v1/sync/authenticate, user: edge, total time: 0:00:00.276, processing time: 0:00:00.276, SQL time: 0:00:00.009 (count: 20), source: 199.91.140.61 , type:rest, method:GET, api_name:sn_edge_encryption/sync, resource:sn_edge_encryption/v1/sync/authenticate, version:v1, user_id:4fd4b095dbdcb200d5cff2131f961928, response_status:500

 

Release

Starting in the Madrid Release.

Environment

This will most likely be seen on instances that have been restored from a backup or following a clone.

Cause

At System Definition -> Certificates there may be one or multiple Certificates named "sn_edge_instance_selfcert"

The certificates(s) will either not have an attachment named "sn_edge_instance_selfcert.p12" or that attachment may be from a different instance due to cloning or restore from backup.

The missing attachment or incorrect attachment will cause the proxy not to be able to Authenticate.

From the node log above you will see reference to the "sn_edge_instance_selfcert" in the error messaging, in this case the attachment was missing in the certificate:

SEVERE *** ERROR *** Attachment is missing for certificate sn_edge_instance_selfcert - 8d80f1a6dbcc3b409063f4eabf961922

Resolution

(1) Go to System Definition -> Certificates 

(2) Delete all certificates there with the name "sn_edge_instance_selfcert"

(3) Go back to the Edge Encryption Configuration -> Proxies -> All -> select the proxy to be Authenticated and select "Authenticate" if there are no other issues the proxy should now go to "Authenticated".  A new corrected "sn_edge_instance_selfcert" certificate will be created with the proper attachment.

(4) To cover the clone case for future clones:  For what Edge Encryption related sys_certificate records to exclude during a clone see KB0696135

Article Information

Last Updated:2019-08-19 13:09:42
Published:2019-08-19