722 views

Description

The Edge Encryption proxy faces extreme slowness at the client browser.

Steps to Reproduce

 

  1. Start an Edge Encryption proxy.
  2. Connect to an instance.
  3. Simulate users going through the proxy and using connection-intensive pages such as Service Portal and Knowledge.
    Note that after a certain threshold, response times become extremely slow at the browser. 
  4. To confirm that you are experiencing this issue:
    1. Turn on additional logging by referring to KB0621289 and activating the "Logging for the Jetty Application Server Client" logging.
    2. In the jettyClient.log logfile, look for the following, specifically "Max requests per destination 1024 exceeded for HttpDestination." If this appears, you are experiencing the issue:
2017-09-25 11:34:36,343 [qtp511631800-158] DEBUG org.eclipse.jetty.client.HttpExchange - Failed HttpExchange@524a6aa9 req=COMPLETED/java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]@23041315 res=COMPLETED/java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]@23041315: req=true/rsp=true java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0] 
2017-09-25 11:34:36,343 [qtp511631800-158] DEBUG org.eclipse.jetty.client.HttpExchange - Aborted before association HttpExchange@524a6aa9 req=COMPLETED/java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]@23041315 res=COMPLETED/java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]@23041315: java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0] 
2017-09-25 11:34:36,372 [qtp511631800-110] DEBUG org.eclipse.jetty.client.HttpDestination - Max queue size 1024 exceeded by HttpRequest[POST /amb/handshake HTTP/1.1]@19569ffe for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0] 
2017-09-25 11:34:36,372 [qtp511631800-110] DEBUG org.eclipse.jetty.client.HttpDestination - Max queue size 1024 exceeded by HttpRequest[GET /sn_edge_encryption_Edge_Encryption_Error.do HTTP/1.1]@19569ffe for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0] 
2017-09-25 11:34:36,372 [qtp511631800-110] DEBUG org.eclipse.jetty.client.HttpExchange - Failed HttpExchange@67fab66c req=COMPLETED/java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]@45908156 res=COMPLETED/java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]@45908156: req=true/rsp=true java.util.concurrent.RejectedExecutionException: Max requests per destination 1024 exceeded for HttpDestination[https://proxy.url.com]77b8c8da,queue=1024,pool=DuplexConnectionPool[c=256/256,a=256,i=0]

Workaround

The issue is caused by a bottleneck in Jetty, the web server and java servlet container where the edge proxy runs. The Jetty configuration needs to be changed to provide optimum performance for the traffic levels being presented to the proxies.  

There is no workaround for this issue. The source code fix for this problem must be applied to the edge encryption proxies following the standard edge encryption proxy upgrade procedure to provide relief. 

Initially, Jetty was configured to only allow 256 concurrent connections to a single IP address. The correction for this problem sets the allowed number of concurrent connections to a default of 10,000. This can also be configured in the edgeencryption.properties file using the edgeencryption.proxy.max.connections property, for example:

edgeencryption.proxy.max.connections = 15000

At this time, we do not believe that the default of 10,000 needs to be modified.

Also, the number of Jetty selectors, threads that manage events on connected sockets, are set based on a formula recommended by Jetty:

number of processor cores where the proxy is running divided by 2, if there is only one core the number of selectors is set to 1.

This can also be configured in the edgeencryption.properties file by the use of the edgeencryption.proxy.selector.count property, for example:

edgeencryption.proxy.selector.count = 2

At this time, we do not believe that the calculated default needs to be modified.


Related Problem: PRB916539

Seen In

Helsinki

Fixed In

Helsinki Patch 10
Helsinki Patch 4 Hot Fix 5
Istanbul Patch 5
Istanbul Patch 6
Jakarta

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2017-09-28 17:26:07
Published:2017-09-29