Test Case Categories
We want to keep you updated on the status of your Test Plans and Test Cases with more context than just a simple Pass
or Fail
result.
Each Test Case Result will have a detailed format, including a Status, Category, and Reason. This will help you focus your efforts and identify the reason behind test results at a higher level.
For example, we have a Error
status that makes it very clear to you which Test Cases require your analysis.
Statusesβ
Statuses show which state the Test Case is in. For example, if a Test Case is currently being executed, it will show the Running status!
Passedβ
β The test completed execution and passed!
Failedβ
π« The test completed execution but failed. The Category and Category Reason help describe why before diving deeper into details like the Trace Viewer.
Qualiti Errorβ
π¨ The test ran into an issue on Qualiti's side. The Category and Category Reason can help clarify the issue, but if you need further assistance, feel free to contact our support team.
Errorβ
π The test completed execution and failed. This could be a Qualiti issue or a customer issue, and requires your analysis to determine the cause.
Queuedβ
β³ The test is waiting to be executed or currently running.
Skippedβ
β© The test will not be executed, but still count towards Test Coverage as Skipped Test Coverage.
Deletedβ
π The test is removed from your Test Library and will not be executed.
Categoriesβ
Test Results are categorized into specific buckets, which helps in identifying the root cause and implementing a more focused and effective solution. The following categories and reasons provide insights into why a test did not pass.
Failedβ
This means the test case failed, and Qualiti was able to determine it was due to an environment or application issue.
- Data Issue
- Failed On Action
- Element Not Found
- Login
- Page Timed Out
Errorβ
This means Qualiti was unable to determine whether the test passed or failed. You'll need to analyze the result to decide what action to take. For example, creating a bug in your project management tool or updating a test case step.
- Network Error
- Element Not Found
- Stale Element
- Navigation
- Element Not Interactable
- Unreliable Selector
- Failed On Action
- Failed On Assert
- Failed On Locator
- Login
π Not sure what the difference is between a Failure and an Error? Click here to learn more.
Qualiti Errorβ
This means that an issue happened within Qualiti's system and has been flagged for our team to investigate and resolve. We recommend running the test again.
- Runtime Error
- Unsupported Event
- Browser Setup
- Ignored
- Infrastructure Error
- Login
- Result Parser
- Results
- Step Finalization
- Step Initialization
- Test Run Setup
- Test Run Teardown
- Timed Out
- Validation
Understanding Failures vs Errors in Test Resultsβ
When reviewing your test results, it's important to distinguish between Failures and Errors, as they represent different types of issues:
-
Failure: A failure means the system successfully ran the test but encountered a problem that caused the test to fail. This usually points to an issue with the application itselfβfor example, an unexpected UI change or incorrect output. The test executed as expected, but the result did not meet the defined assertions.
-
Error: An error indicates that the test could not be executed due to an unexpected problem. This could be due to setup issues, missing elements, network problems, or exceptions in the test code. In these cases, the system is unable to complete the test steps at all.
Understanding this difference can help you quickly identify whether an issue lies within your application (Failure) or within your test environment or configuration (Error).