Notifications

132 views

Description

ATF Tests fails to report test result when excessive console logs are generated

Steps to Reproduce

1) Navigate to the Automated Test Framework (ATF) > Properties module and enable the following checkboxes:
- Under "Test/Test Suite Properties": "Enable test/test suite execution"
- Under "Test Debugging Properties": "Enable additional debugging"
2) Click "Save".
3) Navigate to the "System UI > UI Pages" module and click "New".
4) Input any name. Copy the following in the "HTML" field:
<?xml version="1.0" encoding="utf-8" ?>
<j:jelly trim="false" xmlns:j="jelly:core" xmlns:g="glide" xmlns:j2="null" xmlns:g2="null">
Hello ATF!
</j:jelly>
5) Copy the following in the "Client script" field:
// Wait mechanism ignores setTimeouts with matching stack traces so call this recursively to "trick" it into thinking they are different
function foo(i) {
if (i < 200) {
setTimeout(function() {
// Intentionally blank
}, 1000);
foo(i + 1);
}
}
foo(0);
6) Click "Submit".
7) Navigate to the "Automated Test Framework (ATF) > Tests" module and click "New".
8) Input any value for "Name" and press "Save".
9) Click "Add Test Step", navigate to "Form" category and select "Open an Existing Record".
10) Set Table to "UI Page [sys_ui_page]" and Record to the UI page created in Steps 1-5 and click "Submit".
11) Click "Add Test Step", navigate to "Form" category and select "Click a UI Action".
12) Set Table to "UI Page [sys_ui_page]" and UI action to "Try It", then click "Submit".
13) Click "Add Test Step", navigate to "Custom UI" category and select "Assert Text on Page (Custom UI)".
14) Set "Text" to "Hello ATF!" and "Assert type" to "Text is on page".
15) Click "Run Test", select "Start a new test runner" then click "Run Test".

Observe the test seems to run in the client test runner, but gets stuck with a full progress bar at "Finished test execution, reporting result".
In the browser console, the message "ReportUITestProgress.reportBatchResult: failed X time(s) to report batch result, http status 0, trying again" is reported repeatedly until the batch times out after 10 minutes.
The test eventually ends in failure with output "The test timed out because the max execution time (600 seconds) was reached while executing steps 1 - 3".

Workaround

Terminology

- UI batch: A sequence of continuous UI Step types, or Step Configurations where its Step environment is "UI". Example UI steps: Form > Set Field Values, Service Catalog > Order catalog Item. To break up a batch, use a server step in between two UI steps.

- Batch order constraint: Step Configurations have a column that defines where in a UI or Server batch they can be run. Steps with batch order constraint "Start a batch execution" tend to navigate to a page without any dependency on the prior state of the browser, other than that a Client Test Runner is available to run that UI batch.


The goal of this workaround is to decrease the amount of console logs and reporting data being sent from the Client Test Runner back to the server in a single AJAX call. The below solutions attempt to decrease the data being generated per test UI batch.

There are two recommendations:

1) Disable ATF Debugging

Disable Debugging to decrease the number of console logs that the Client Test Runner reports back to the server:

Directions

  1. Navigate to the Automated Test Framework (ATF) > Properties module
  2. Under "Test Debugging Properties" -> "Enables additional debugging" -> uncheck the checkbox, click "Save".
  3. Try running problematic tests again. If they continuously report success without issue, you can skip other workaround recommendations.


2) Break up problematic tests that have long continuous UI batches so that they send less data back to the server in a single reported test progress AJAX call.

This can be accomplished by adding no-operation Server environment steps in the middle of long sequences of UI environment steps.

The UI steps that you can add a Server step before, without interrupting a dependent sequence of UI behavior, are any that have a Step configuration > Batch order constraint of "Start a batch execution". To see a list of these, navigate to Automated Test Framework (ATF) > Administration > Step Configurations, then filter by Batch order constraint is "Start a batch execution" and Step environment is "UI".

Directions for tests that demonstrate this defect:

  1. Find continuous UI steps in the test, for example: more than 10 steps in a row. Break them up where possible with a Server step Server > Run Server Side Script but with no embedded script. This Server step can be placed before any UI step with a Batch Order Constraint fo "Start a batch execution".
  2. Run the test and confirm it still passes without introducing a breaking change due to a dependency between the two steps that were separated by the Server step. Confirm the test passes.
  3. repeat for all problematic tests



Related Problem: PRB1388207

Seen In

SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - PA Premium Integration - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - IRM - Vendor Risk Management - Madrid 2019 Q1
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response UI Patch - London 2019 Q2 v.6.2.3
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - SIR - Threat intelligence - New York 2019 Q3
SR - SIR - VirusTotal Integration - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Intended Fix Version

Quebec

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-09-23 13:21:30
Published:2020-08-06