ServiceNow internal monitoring system alerts when it detects that your instance is down
At this time you may also experience that you are unable to access the instance and/or the response time is extremely slow.
The nodes keep going out of memory as well.
Transaction logs show the error:
SEVERE *** ERROR *** OutOfMemoryError: Java heap space
Upon investigation it was found that this mid.eccq.max_payload_size was added and the value was set to 300mb instead of 20 mb
Actual payload that was being received for the SOAP call was 178 MB
As an application, it is not possible to handle this size (178 MB). SOAP payload is xml. If payload is to be processed, it will need to parse the XML. The memory required to parse XMLs are generally 10 - 20x the payload size.
Since the application server node cannot have more than 2GB heap, this will result in such memory issues.
Temporary Relief Measure
Please revert the value for glide.soap.max_inbound_content_length property from 300 MB to the default 70 MB
The SOAP payload needs to be under 300MB heap usage which means the payload needs to be 15MB or less.
The SOAP requests can be broken into chunks so that the data is brought back in batches.