6010 views

Description

You have configured a POP3 or IMAP email account on the instance that uses a Microsoft Exchange Server, Server = outlook.office365.com in the email account record.

By observing the Exchange Server email backlog, it is noticed that emails are not being read by the instance even though the email account is active and the connection test passes on the instance.  As a result, the email backlog on the Exchange Server continues to grow larger.

The way the email reader on the instance works (reading in emails using pop3 or IMAP protocols) is that the instance reads 20 emails (by default or a different number as defined in the system property com.glide.email.max_read) at a cycle.  Once those emails are read into the instance it sends a delete command to the email server to delete those read emails from the email server, then the next 20 are read and deletes are sent for those and so on until all of the emails are processed.

If the delete does not work on the email server the instance will keep reading the same 20 emails over and over again.

To debug the issue set the glide.pop3.debug or glide.imap.debug (depending on if the account uses pop3 or imap protocol) to true.

With glide.pop3.debug or glide.imap.debug system properties turned on - see the email (identified by the Message-ID) get deleted when it is read in the wrapper_<date>.log :

INFO | jvm 2 | 2018/06/01 09:49:28.591 | Subject: Important Email
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Thread-Topic: Important Email
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Thread-Index: Aalqu8jcxQ==
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Date: Fri, 1 Jun 2018 09:47:36 -0700
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Message-ID:
INFO | jvm 2 | 2018/06/01 09:49:28.591 | <SN50102MB309B2AB554843DE444A9620@SM44354.prod.sn..com>
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Accept-Language: en-US
INFO | jvm 2 | 2018/06/01 09:49:28.591 | Content-Language: en-US
INFO | jvm 2 | 2018/06/01 09:49:28.591 | X-MS-Exchange-Organization-AuthAs: Internal
INFO | jvm 2 | 2018/06/01 09:49:28.591 | X-MS-Exchange-Organization-AuthMechanism: 04
...
...
...
INFO | jvm 2 | 2018/06/01 09:49:28.604 | JZHgbMjkFgWbcxHHBzUkvhLS5rMQSRyFWmMrN5h3FhnHPoMngcUUUXAgi8I6aJriVhK7XFsYZQzA

INFO | jvm 2 | 2018/06/01 09:49:28.604 | hhxzjHXgV0sKLFBGijCqoUD2AoopMZ//2Q==
INFO | jvm 2 | 2018/06/01 09:49:28.604 |
INFO | jvm 2 | 2018/06/01 09:49:28.604 | --_003_DM5PR0102MB3463C957BF09B23745E3F344A9620DM5PR0102MB3463_--
INFO | jvm 2 | 2018/06/01 09:49:28.604 | .
INFO | jvm 2 | 2018/06/01 09:49:28.705 | NOOP
INFO | jvm 2 | 2018/06/01 09:49:28.805 | +OK
INFO | jvm 2 | 2018/06/01 09:49:28.805 | DELE 1
INFO | jvm 2 | 2018/06/01 09:49:28.905 | +OK
INFO | jvm 2 | 2018/06/01 09:49:28.905 | QUIT
INFO | jvm 2 | 2018/06/01 09:49:29.206 | +OK Microsoft Exchange Server POP3 server signing off.

The message is deleted with the DELE 1 command and there is an acknowledgment with the +OK response from the email server, but in reality, the email is not deleted from the Microsoft Exchange email server itself.

So when the email reader runs again it downloads the same email that is already read and processed, the following will be seen in the system node log (again with glide.pop3.debug or glide.imap.debug system properties turned on) indicating that an email with the same Message-ID is already on the instance:

2018-06-01 09:51:29 (695) worker.3 worker.3 Message UID=‘123456' Message-ID='<SN50102MB309B2AB554843DE444A9620@SM44354.prod.sn..com>' was previously read and will be ignored.

If you look on the instance for the sys_email records you will find an email already exists on the instance with the Message-ID='<SN50102MB309B2AB554843DE444A9620@SM44354.prod.sn..com>'

Release or Environment

This applies to any ServiceNow release.

Cause

The failure to delete the message from the Microsoft email server may be due to the email account being on “Litigation Hold" which in itself is not stopping the deletes from happening on the email server, but while the account is on "Litigation Hold" there can only be 100GB of deleted emails allowed for the account.  If this 100GB threshold has been reached while on “Litigation Hold" the delete commands sent by the ServiceNow instance will not be honored by the Microsoft email server resulting in this issue.

Another reason may be that the mailbox user has not set up an archive to move emails there automatically.

A third cause of this issue that has been seen is where a customer had a Deleted Items sub-folder off the main inbox (the inbox that the Email Reader was retrieving emails from), and their mail admin had manually started a long-running delete process on that sub-folder. Their Deleted Items sub-folder had had tens of thousands of emails deleted from it, and it appeared that the fact that this process was still running was preventing deletions being effective on the parent inbox. See Alternative/Workaround Solution below.

Even though you see the positive +OKs from the Microsoft email server for the delete commands (DELE) sent by the instance in the wrapper_<date>.log, the emails are not actually getting deleted from the Microsoft email server.

Resolution

This issue cannot be fixed on the ServiceNow side.

Please consult the knowledge documents below provided by Microsoft support related to “Litigation Hold" and setting up archiving. If these documents do not resolve the issue, please contact Microsoft support for further assistance:





Alternative/Workaround Solutions
(1) A workaround that might help in some circumstances is to rename the mailbox and create a new one with the original name. For example renaming the mailbox support@<customer.com> to support2@<customer.com>, and then re-creating a brand-new mailbox support@<customer.com> with the same password as before. This should also mean that no changes are required on the ServiceNow side (not even in Email Accounts).

This worked in a situation where a long-running delete on a sub-folder of the original mailbox (support@<customer.com>) was preventing deletes from being effective. This may not work where the cause of the issue is Litigation Hold.

(2) Create and set the system property com.glide.email.max_read to a larger value than 20, say 50 or 100 or 500.  This will allow more emails to be read into the instance.  However, this will not resolve the issue since unread emails will continue to build up until the new limit for com.glide.email.max_read is reached again, when this happens email reading will be stopped again.  Once the issue with the account is resolved please delete the com.glide.email.max_read property or set it back to the default value of 20.

Article Information

Last Updated:2020-11-18 06:22:14
Published:2020-11-18