153 views

Symptoms


You have configured a POP3 or IMAP email account on the instance that uses a Microsoft Exchange Server, Server = outlook.office365.com in the email account record.

By observing the Exchange Server email backlog, it is noticed that emails are not being read by the instance even though the email account is active and the connection test passes on the instance.  As a result the email backlog on the Exchange Server continues to grow larger.

The way the email reader on the instance works (reading in emails using pop3 or imap protocols) is that the instance reads 20 emails (by default or a different number as defined in the system property com.glide.email.max_read) at a cycle.  Once those emails are read into the instance it sends a delete command to the email server to delete those read emails from the email server, then the next 20 are read and deletes are sent for those and so on until all of the emails are processed.
 
If the delete does not work on the email server the instance will keep reading the same 20 emails over and over again.

To debug the issue set the glide.pop3.debug or glide.imap.debug (depending on if the account uses pop3 or imap protocol) to true.

With glide.pop3.debug or glide.imap.debug system properties turned on - see the email (identified by the Message-ID) get deleted when it is read in the wrapper_<date>.log :
 
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Subject: Important Email
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Thread-Topic: Important Email
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Thread-Index: Aalqu8jcxQ==
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Date: Fri, 1 Jun 2018 09:47:36 -0700
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Message-ID:
INFO | jvm 2 | 2018/06/01 09:49:28.591 | <SN50102MB309B2AB554843DE444A9620@SM44354.prod.sn..com>
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Accept-Language: en-US
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Content-Language: en-US
INFO | jvm 2 | 2018/06/01 09:49:28.591 | X-MS-Exchange-Organization-AuthAs: Internal
INFO | jvm 2 | 2018/06/01 09:49:28.591 | X-MS-Exchange-Organization-AuthMechanism: 04
...
...
...
INFO | jvm 2 | 2018/06/01 09:49:28.604 | JZHgbMjkFgWbcxHHBzUkvhLS5rMQSRyFWmMrN5h3FhnHPoMngcUUUXAgi8I6aJriVhK7XFsYZQzA
 
INFO | jvm 2 | 2018/06/01 09:49:28.604 | hhxzjHXgV0sKLFBGijCqoUD2AoopMZ//2Q==
INFO | jvm 2 | 2018/06/01 09:49:28.604 |
INFO | jvm 2 | 2018/06/01 09:49:28.604 | --_003_DM5PR0102MB3463C957BF09B23745E3F344A9620DM5PR0102MB3463_--
INFO | jvm 2 | 2018/06/01 09:49:28.604 | .
INFO | jvm 2 | 2018/06/01 09:49:28.705 | NOOP
INFO | jvm 2 | 2018/06/01 09:49:28.805 | +OK
INFO | jvm 2 | 2018/06/01 09:49:28.805 | DELE 1
INFO | jvm 2 | 2018/06/01 09:49:28.905 | +OK
INFO | jvm 2 | 2018/06/01 09:49:28.905 | QUIT
INFO | jvm 2 | 2018/06/01 09:49:29.206 | +OK Microsoft Exchange Server POP3 server signing off.
 
The message is deleted with the DELE 1 command and there is an acknowledgment with the +OK response from the email server, but in reality the email is not deleted from the Microsoft Exchange email server itself.
 
So when the email reader runs again it downloads the same email that is has already read and processed, the following will be seen in the system node log (again with glide.pop3.debug or glide.imap.debug system properties turned on) indicating that an email with the same Message-ID is already on the instance:
 
2018-06-01 09:51:29 (695) worker.3 worker.3 Message UID=‘123456' Message-ID='<SN50102MB309B2AB554843DE444A9620@SM44354.prod.sn..com>' was previously read and will be ignored.
 
If you look on the instance for the sys_email records you will find an email already exists on the instance with the Message-ID='<SN50102MB309B2AB554843DE444A9620@SM44354.prod.sn..com>'
 

Release


This applies to any ServiceNow release.

 

Cause


The failure to delete the message from the Microsoft email server may be due to the email account being on “Litigation Hold" which in itself is not stopping the deletes from happening on the email server, but while the account is on "Litigation Hold" there can only be 100GB of deleted emails allowed for the account.  If this 100GB threshold has been reached while on “Litigation Hold" the delete commands sent by the ServiceNow instance will not be honored by the Microsoft email server resulting in this issue.
 
Even though you see the positive +OKs from the Microsoft email server for the delete commands (DELE) sent by the instance in the wrapper_<date>.log, the emails are not actually getting deleted from the Microsoft email server.
 

Resolution


This issue cannot be fixed on the ServiceNow side. Please consult the knowledge documents below provided by Microsoft support related to “Litigation Hold". If these documents do not resolve the issue, please contact Microsoft support for further assistance:
 
https://support.office.com/en-us/article/increase-the-recoverable-items-quota-for-mailboxes-on-hold-a8bdcbdd-9298-462f-b889-df26037a990c
 
https://technet.microsoft.com/en-us/library/dn743673(v=exchg.160).aspx
 
https://support.office.com/en-us/article/Delete-items-in-the-Recoverable-Items-folder-of-cloud-based-mailboxes-on-hold-Admin-Help-a85e1c87-a48e-4715-bfa9-d5275cde67b0?ui=en-US&rs=en-US&ad=US
 
 

Article Information

Last Updated:2018-06-11 16:07:07
Published:2018-06-11