This new feature is called the ATF Client Error Whitelist and will allow manual or automatic addition of specific client errors to a Whitelist and have these errors skipped during execution of ATF tests or Test Suites.
There are actually two methods of adding specific client-side errors to this Whitelist; automatic and manual. Both methods will be discussed in this article as well as information regarding as to the purpose of these record in the ATF system.
The automatic method of adding client errors to the ATF Client Error Whitelist is the method that will be used the vast majority of the time, but on occasion, the need may arise to manually add such a record. Thus, we will first review the steps used to add an ATF Whitelist error manually.
Adding Errors Manually to the Whitelist
ATF Whitelist client error records can be added to the ATF manually by specifying various information about that error. The following step will detail how this can be done:
Log into the instance with an account having admin, atf_admin or atf_test_designer rights to the instance.
Browse to the following location on the instance: Automated Test Framework -> Run -> Whitelisted Client Errors.
A list of current records (if any) in the Whitelisted Error table will appear. Click the New button to create a new record in this table.
The following fields should then be filled out on this form:
Report Level: This field is a pull down menu that should be set to how you would prefer this particular error to be reported in the Test Results for any test for which it is found. Currently, the following are the options that can be selected for this field:
- Warning - Step & Test will report Success with Warning(s) - This option will cause the test to not halt on this error, but will log the error as a Warning in the test step results, with the test step encountering the error showing a result code of Success with Warning(s).
- Ignored - Step & Test will report Success - This option will also cause the test to not halt upon encountering the error. The test step encountering this Whitelisted client error will report a status code of Success, however the test log for this test will contain a record showing the client error and show that the error has been Ignored.
Note that if the need arises to again halt the test on this error and report a failed test, this Whitelisted error record will either need to be deleted from the instance or the status of the Whitelisted Error set to not be active.
Active: This checkbox should be checked to ensure that this Whitelisted Client Error will be consulted by the ATF in the event that this error type is encountered during run of each test step. If this step is set to not be active this record will be ignored and the error will still reported and will cause the ATF test to halt as would normally be expected.
Error Message: This is the specific error message you intend to have added to the Client Error Whitelist. This should normally be the error message that is generated when this specific error is encountered. The system actually uses a "contains" operator when comparing the Error Message field with the actual error generated. Thus, if a client-side error message is encountered in running the test and the text of that error message contains a substring of text that matches the text in this field, this Client Error Whitelist will be applied to that record.
Description: This field should contain a brief description of the error that is being added to the Whitelist.
Once all the fields have been filled out a requested, click the Submit button to add the new Whitelist Client Error record to the ATF system.
Adding Errors Automatically to the WhiteList
The more common method to add WhiteList Client Error records to the instance is to add them directly from the Test Record, the Test Result record or the Test Step Result record.
Adding the step in this manner, will of course require that the test have been run at least one time with this error occurring for that test.
To add the WhiteList Error records directly from the Test result. Browse to the Test Results record for the test which had failed due to the client-side error. Scroll to the Related Links associated to that record and select either the link titled "Add all client errors to warning list" or the link titled "Add all client errors to ignored list".
After a moment any errors encountered by the test (before the test halted) will be added to the Warning or Ignore list, based on which option was selected, and a message will appear indicating the client error records have been added.
To add the WhiteList Error Records directly from the Test Step Result. Similar to adding the Client Error Whitelist record from the test result, the record can also be added from the Test Step Result record. To do this, browse to the specific Test Step Result which had failed to the browser error. From the Related list, select either the link with the title "Add all client errors to warning list" or the link titled "Add all client errors to ignored list". After a few moments, any error records encountered from this step will be added to the Client Error Whitelist and a message will appear indicating the records have been added.
To add the Whitelist Error from the Test Result Item record. To add the Client Error Whitelist record directly from the Test Result Item, first browse to the specific Test Result Item which encountered the specific error. Locate the Related Links associated to this record and select either the option "Add client error to warning list" or the option "Add client error to ignored list". After selecting this option the particular error message encountered on this step will be added to the appropriate list and a message will appear indicating the record has been added to the list.
As mentioned previously, adding the error to the Warning list will cause the error to be logged as a Warning, and the test will continue, but be marked as Successful with Warnings upon completion due to this error. If the error is instead added to the Ignored list, the test will also continue showing as Successful, but a record in the test results indicating the error had been encountered but ignored.
As an example, say that in performance of an ATF test, the test had encountered a simple client side script error during it's run. This error would thus cause the test to halt, reporting failure of the test. It would also generate a Failure on the test step itself.
In most cases, the tester would want to fix any such client-side errors encountered in associated objects, however there are occasions when the error cannot or should not be fixed. Thus, for this example, the tester would add the error encountered to the Client Error Whitelist in the Warning list. In that case, the test would complete reporting a status of Success with Warning(s).
In addition the individual test steps which encountered that particular error would be marked as Warning.
However, if instead of adding the client script error to the Warning list, the client script error was added to the Ignore list, the test would complete with a status of Success.
The individual test step which had failed would still appear in the test steps as Ignored and would not cause the test to halt due to that error.
There are a few very important notes to keep in mind when working with the Client Error Whitelist:
- Adding a specific error message to this list will cause the ATF system to take the appropriate action for all occurrences of that error on any ATF Test or Test Suite for which it ran. This will thus allow the test to continue beyond the error encountered. However, this does not guarantee that the test will not fail or halt due to other errors encountered in the same test or suite.
- In most cases, in the event that a client-side error is detected in an ATF test, the best option is to attempt to correct the error, to not only allow the test itself to complete successfully, but also to ensure that proper operation of the functionality of the instance. However, in certain cases the error may not be correctable due to a number of causes, in which the client side error should then be added to the Client Error Whitelist.
- If a Whitelisted error needs to again cause execution of a test to fail, the Client Error Whitelist record corresponding to that error should either be removed (deleted) from the instance or the Active checkbox for that record set to not be active.