Description

---

It is possible that jobs running in MID Servers get stuck, and then block one of the worker threads, eventually leading to a MID Server with no threads left available to run anything. This can happen due to poor design of custom JavascriptProbe jobs, or there have been some product defects that also leave stuck threads behind after the job has finished.
A MID Sever restart clears stuck threads. Until your causes of stuck threads are fixed, one possible workaround is to have a scheduled script run periodically to tell the MID Server(s) to restart themselves. 

Procedure

---

1. Open a list of all MID Servers: In the navigation MID Server -> Servers. (ecc_agent table)
2. Filter the list down to only the MID Servers(s) you need to be restarting. e.g. Name Contains 'Disco'.
3. Add an extra condition for Status IS Up, so that we only restart MID Servers that are running.
4. Right click the blue filter line and 'Copy query'. This will give you the Encoded Query String for this list filter that we will use in the script. e.g. "nameLIKEdisco^status=Up"
   
   
   
   
5. Open a new Scheduled Script record (sysauto_script): System Definition -> Scheduled Jobs, click New, click Automatically run a script of your choosing.
6. Fill in Name: (custom) Restart MID Servers to clear stuck threads
7. Fill in the schedule fields. e.g. Run: Weekly, Day: Sunday, Time: 07:00:00. Pick a time when no jobs are likely to be running in your MID Servers.
8. Paste the following script into the Run this script field:
   
   

   // scheduled script to regularly restart MID Servers to clear out stuck threads (KB0692080)
   var midGr = new GlideRecord('ecc_agent');
   midGr.addEncodedQuery('<paste query string here>'); // This query string limits which MID Servers are restarted.
   midGr.query();
   while(midGr.next()) {
   	var agent_name = midGr.name.replace(/'/g, "\\'");
   	var midmanage = new MIDServerManage();
   	midmanage.restartService(agent_name); // This line writes the restart ecc_queue output to the mid server.
   	gs.info('(custom) Restart MID Servers to clear stuck threads job: Restarting MID Server: ' + agent_name);
   }
   

9. Paste your query string copied earlier into the addEncodedQuery function highlighted above. The line will then look like e.g.
   
   

   midGr.addEncodedQuery('nameLIKEdisco^status=Up'); // This query string limits which MID Servers are restarted.
   

10. Submit. The script will now run on the next scheduled time.
11. To test this script, or run it on demand, use the Execute Now button.