Test Environment Basics
Test Run Lifecycle
A Test Run at Qualiti goes through 3 phases:
- Environment Setup (optional, but recommended)
- Test Plan Execution
- Environment Teardown (optional)
Although the Setup and Teardown phases are “optional”, it is recommended to use at least the Setup Phase to ensure that your System Under Test (SUT) and Environment Under Test (EUT) are in a state that’s ready to be tested.
Test Plan Execution
A Test Plan is a suite or collection of tests to be executed. Qualiti generates and maintains tests automatically, but you may want a specific Test Plan configuration for your different SUTs. For example, a Test Plan for the Staging environment of your e-commerce app.
This phase is about running the tests, getting results, and ultimately reporting them - gathering screenshots, videos, bugs, etc.
A Test Run consists of executing one or more Test Plans.
Environment Setup
The Setup phase is crucial because it gives you the opportunity to get your systems and overall environment into a stable, testable state. Automation works best when tests and results are deterministic - meaning that you can repeat the same test and get the same result.
Qualiti provides a Pre-Hook which allows you to trigger a Test Plan Execution once your SUT and EUT are ready!
Some use cases include:
- Resetting Test Data in the Environment
- Triggering Test Plan Execution once the STAGE Environment is fully deployed
Environment Teardown
Similar to the Setup phase, Qualiti provides a Post-Hook so you can trigger things once a Test Plan Execution is completed.
Some use cases include:
- Scrubbing a database to remove all the data that was generated during test execution
- Approving a Pull Request or Merge Request in your CI Pipeline
- Sending reports to the appropriate Developers, Testers, and/or stakeholders
Recommended Practices
Each organization has a unique approach to tech, but the fundamental concepts are similar. In this section, we will discuss best practices that we recommend to maximize the benefits of Qualiti and enhance your internal quality.
Test Environments
Also known as the Environment Under Test (EUT), the Test Environment is any environment where your tests and experiments are being executed.
Let's take a common Development Progression: Local -> Dev -> Stage -> Prod
Local
This is where the developer starts programming. The “Environment” in this case is usually their laptop - its resources, CPU, memory, etc. If they’re using something like Docker, those resources are still from their LOCAL machine.
Developing “in the cloud” is becoming more common because of solutions like Gitpod. In those cases, developers start writing code and build things directly into DEV.
Types of tests that exist here would include:
- Unit Tests
- Component Tests
- Contract Tests
Dev
Once they've built and tested their code locally, they deploy it to DEV where multiple features and services are integrated. In the diagram above, Dev A and Dev B are either pairing on a feature or have features that need to be integrated together.
Note how Dev A and Dev B have their own environment with dedicated services and databases. Dev C also has its own isolated environment as well.
Now that multiple services and systems are integrated, you can perform higher-level tests including:
- UI Tests
- API Tests
- Database Tests
- End-to-end (E2E) Tests
Some teams will have their Pull Requests (PR) or Merge Requests (MR) to the main branch happen here. This is where having good quality gates - including Test Automation - in your Continuous Integration (CI) pipelines matters.
Stage
After integrations, flows, and larger systems have been tested in DEV, they deploy to STAGE. This environment does its best to look and feel like PRODUCTION. Because very few companies want to spend the same amount on STAGE as they do in PROD, STAGE usually has fewer resources, weaker computing power, and less storage.
In other words, STAGE is smaller and different than PROD. Just because tests and experiments pass in STAGE does not mean they will pass in PROD.
As you can see from the diagram above, this is where all of the features and code come together into a singular environment - the same way it would be in PROD. With the entire system now intact, your tests, demos, and experiments will be more “accurate”.
Types of tests that exist here would include:
- Usability Tests
- Performance Tests
- Larger E2E Tests
Production
Finally, once STAGE is stable, validated, and ready, the team deploys it to PROD where your users get the value of all your hard work!
Testing, observing, and operating production are important, but out-of-scope for this doc.
Types of tests that exist here would include:
- Smoke Tests
- A/B Testing
- Monitoring
- Progressive Deployments
Dedicated Test Environment for Qualiti
Although you can make any environment within DEV, STAGE, or PROD a Test Environment, it’s recommended that you create a dedicated environment specifically for Qualiti’s tests. As you saw above, there may be an environment where developers are actively working, and their changes - or the changes for the tests - could lead to unexpected behaviors or failing tests that result in false positives or false negatives.
Qualiti will execute tests on whichever environment you give it, so make a separate Test Environment with a copy of the services, databases, and any other resources you’d like to have tested!