Skip to main content

Test Case Categories

We want to keep you updated on the status of your Test Plans and Test Cases with more context than just a simple Pass or Fail result.

Each Test Case Result will have a detailed format, including a Status, Category, and Reason. This will help you focus your efforts and identify the reason behind test results at a higher level.

For example, we have an Error status that makes it very clear to you which Test Cases require your analysis.

Statuses​

Statuses show which state the Test Case is in. For example, if a Test Case is currently being executed, it will show the Running status!

Passed​

✅ The test completed execution and passed!

Failed​

🚫 The test completed execution but failed. The Category and Category Reason help describe why before diving deeper into details like the Trace Viewer.

Needs Review​

⚠️ The test requires your attention before it can be considered complete. This could be due to incomplete setup, configuration issues, or results that need human validation.

Qualiti Error​

🐨 The test ran into an issue on Qualiti's side. The Category and Category Reason can help clarify the issue, but if you need further assistance, feel free to contact our support team.

Error​

👀 The test completed execution and failed. This could be a Qualiti issue or a customer issue, and requires your analysis to determine the cause.

Queued​

⏳ The test is waiting to be executed or currently running.

Skipped​

⏩ The test will not be executed, but will still count towards Test Coverage as Skipped Test Coverage.

Deleted​

🙈 The test is removed from your Test Library and will not be executed.

Categories​

Test Results are categorized into specific buckets, which helps in identifying the root cause and implementing a more focused and effective solution. The following categories and reasons provide insights into why a test did not pass.

Failed​

This means the test case failed, and Qualiti was able to determine it was due to an environment or application issue.

  • Data Issue
  • Failed On Action
  • Element Not Found
  • Login
  • Page Timed Out

Needs Review​

This means your test needs some attention before it can run successfully or be considered complete. Here are the most common reasons:

  • Step Generation Issues: The test steps weren't fully or correctly generated and need to be completed or fixed
  • Manual Review Required: The test may need manual editing, clarification, or approval before it can be considered ready for automated execution
  • Profile or Configuration Mismatch: The test may have been created or run with an incorrect user profile or configuration, which needs to be reviewed and corrected
  • Test Results Require Attention: The test may have encountered issues, errors, or unexpected results that require human review

Error​

This means Qualiti was unable to determine whether the test passed or failed. You'll need to analyze the result to decide what action to take. For example, creating a bug in your project management tool or updating a test case step.

  • Network Error
  • Element Not Found
  • Stale Element
  • Navigation
  • Element Not Interactable
  • Unreliable Selector
  • Failed On Action
  • Failed On Assert
  • Failed On Locator
  • Login
note

🔍 Not sure what the difference is between a Failure and an Error? Click here to learn more.

Qualiti Error​

This means an issue occurred within Qualiti's system and has been flagged for our team to investigate and resolve. We recommend running the test again.

  • Runtime Error
  • Unsupported Event
  • Browser Setup
  • Ignored
  • Infrastructure Error
  • Login
  • Result Parser
  • Results
  • Step Finalization
  • Step Initialization
  • Test Run Setup
  • Test Run Teardown
  • Timed Out
  • Validation

Understanding Failures vs Errors in Test Results​

When reviewing your test results, it's important to distinguish between Failures and Errors, as they represent different types of issues:

  • Failure: A failure means the system successfully ran the test but encountered a problem that caused the test to fail. This usually points to an issue with the application itself—for example, an unexpected UI change or incorrect output. The test executed as expected, but the result did not meet the defined assertions.

  • Error: An error indicates that the test could not be executed due to an unexpected problem. This could be due to setup issues, missing elements, network problems, or exceptions in the test code. In these cases, the system is unable to complete the test steps at all.

Understanding this difference can help you quickly identify whether an issue lies within your application (Failure) or within your test environment or configuration (Error).