Resetting Test Environment Data
A Clean Environment is a Good Environment
Having a clean environment is NOT required for Qualiti to work, but it is recommended. This doc goes over WHY managing your Test Data is important and shows an example of HOW you can reset your Test Data.
Many automation initiatives fail - especially in testing - because the state of the System Under Test (SUT) and/or the Environment Under Test (EUT) are not well-controlled and deterministic. Having a system that is deterministic means that, given a particular input, the system will always produce the same output while always passing through the same sequence of states. The same is true for the Test Data your system uses.
While testing, it is essential to have a clean environment that does not have any data from previous tests. A clean environment ensures that the test results are accurate and that the testing process is efficient.
Resetting your Test Data is the process of removing all the data from the previous tests and resetting the environment with a clean dataset.
Unclean E-Commerce Example
Let's illustrate an unclean scenario with an e-commerce website that has this cart:
We want to test that we can successfully checkout with a cart that has multiple quantities for multiple items.
With our automated test created and working locally, we start running it in our CI Pipelines:
✅ The test passes the first time it's executed!
❌ The second time it fails with some generic Cypress or Selenium error
🤔 A dev or tester tries it manually and it's working… what gives??
They mark the test as “flaky” and now someone has to debug the test and find the root cause. Eventually, after hours or days of searching, they realize that there is only 1 Sauce Labs Backpack left in inventory for that specific Test Environment.
It's a quick fix, but it wasn't the test that was flaky - it was the environment because it was never reset. Previous test runs affected subsequent ones.
This is common in many organizations because they focus more on automated tests than the underlying systems, environments, and the automation that orchestrates it all.
Qualiti helps a lot with this because the tests are created and maintained automatically using AI, but having deterministic systems and environments is still required because we don't control your Test Environment or Test Data - at least not yet 😉
Steps towards making a Clean Environment
If you don't have this process of resetting data yet, this is the section for you!
Automating the processes of setting up, tearing down, and resetting environments is important so you can speed up development and ensure that you have a clean environment every time. The following steps can help you design, plan, and create this process so you can manually reset data and make clean environments. Then you can work towards automating that process!
We'll continue using the e-commerce demo app as an example.
1. Document your Environment
Before resetting your test environment data, it's important to document your existing environment. This documentation should include all the details of the environment, such as the hardware and software configuration, software installed, network topology, services architecture, and database schema. This documentation can help you to rebuild your environment quickly if anything goes wrong during the resetting process.
We recommend creating a visual diagram to map your infrastructure and architecture.
Example
You already know that there are 3 main parts that make up the application:
- Database - which stores Users and Products (ie inventory, orders, payment methods)
- Web UI - the website our users interact with
- API - the middleware that handles communication between the Database and UI
Your quick diagram looks like this:
It's a good start, but we need the details about these services so we know where and how to reset data. For example, which database are we using?
After talking to the development team, you learn that most of our stuff is hosted in AWS. They create a diagram with the details we're looking for:
It's common for data to be across multiple tables and/or databases! In this case, we have some data in PostgreSQL tables, some in S3, and a CDN to deliver content to the website quickly!
2. Plan the Resetting Process
Resetting your test environment data can be a time-consuming and complex process - especially when done manually. Therefore, it's essential to plan the process in advance to ensure that everything is done in the correct order. The plan should include all the necessary steps, such as shutting down the systems, deleting the data, and configuring the environment for testing.
This also becomes the roadmap needed to automate the environment in the next step
Example
From Step 1, we know that we have data in PostgreSQL and S3, but the Test Data we care about - Users and Products - are in PostgreSQL
So, before executing any tests, we need to reset the data in our PostgreSQL database!
We'll do ths first in DEV, STAGE, AND PROD as we test things out.
For more info about DEV, STAGE, and PROD environments, visit: Test Environments Basics
We have some things we want to make sure we have every time we reset our data for testing:
-
USERS to log in with
- Some with existing carts and orders
- Some with existing payment methods
- Some in different parts of the country or world
- Admins or "super" users
- and any others
-
PRODUCTS in our inventory
- Some with plenty of inventory for every Product option like color and size
- Some with little or no inventory or with options that are sold out
- Some on sale
- Some with mixed reviews, no reviews, only gret reviews, etc
Once we have the backup or "seed" of data we'd like to use every time we reset, we can move to the next step. It's common to have a copy of this seed stored somewhere to make re-using the seed easier
We create our curated seed data into a file called seed_file.dump since we're using PostgreSQL. We can store that file locally or somewhere else like S3. Now we can plan the “Order of Operations” to get that seed data into our database.
PostgreSQL has some tools to help that we'll see below:
(pg_dump, pg_restore, dropdb, and createdb)
An example order of steps could look like this:
-
Wait for the database to be in a stable state
-
COPY or BACKUP the database
pg_dump -U username -Fc database_name > backup_file_name.dump
-
DELETE the database
dropdb --force database_nameImmediately terminates all processes on the database and deletes it
-
CREATE a new database using our curated seed
createdb new_database_name
pg_restore -U username -d new_database_name seed_file.dumpYou may need to configure the database so your existing services start using it
You can keep or discard your backup as needed
-
Start Testing 🎉
3. Use Automated Scripts and Tools
Using automated scripts can make the resetting process faster and more deterministic. Scripts and tools can automate the process of deleting data and configuring the environment, making it easier to reset the environment. Additionally, using automated scripts and tools ensures that the resetting process is consistent every time.
Example
We already designed and planned the resetting process and know which commands and tools we can use! We could wrap those commands into a single bash script that we can execute whenever we need.
Let's call it seed_database.sh:
# 1. Backup the data in case something goes wrong
pg_dump -U username -Fc database_name > backup_file_name.dump
# 2. Delete the existing database
dropdb --force database_name
# 3. Create a new database
createdb new_database_name
# 4. Restore the new database with our seed file
pg_restore -U username -d new_database_name seed_file.dump
Use security best practices when using credentials like username and password
With this automated bash script, we can run it as many times as we want and get the same results every time!
4. Test your Reset Environment
After resetting your Test Data, it is important to test the environment thoroughly to ensure that everything is working as expected. You can run some basic tests to verify that the environment is set up correctly and that all the necessary components are installed and configured.
Once verified, you can continue expanding the automation to include resetting other parts of the environment and not just the data. Eventually, you will have a script that sets up and tears down entire environments at will!
The E-Commerce Example is better now
Our team now has an automated script to reset the data! We can use this for our own internal development and testing, but we can also provide a dedicated Test Environment to Qualiti for better AI-Driven Test Automation!
Eventually, the team will focus on automating these processes for faster and more deterministic environments, but resetting our test environment data is a critical process in the software testing cycle.
Following these recommended practices can help you ensure that the resetting process is efficient, accurate, and effective. By doing so, you can achieve reliable test results and ensure that your software meets the requirements and specifications.