Creating an efficient Test Case Repository – part 2

Welcome back to the two-part blog series on building simple and efficient test repository.

In the first part, we compared our initial and target Test Case Repository structure, offering insights into the repository hierarchical reorganization and Test Case design/creation changes, and explained how Sprint Test Cases are executed and reported on.

The second part will bring into focus organizational procedures needed for maintaining a Test Case Repository by explaining the process of incorporation of Sprint Test Case changes into the Regression Test Suite – in other words, we will talk about Regression Test Suite maintenance.

The repository built in such a way can even serve as a foundation for BDD scenario-driven test automation utilizing some of the popular tools like Selenium (SpecFlow) or Cypress.

The goals of this blogpost about building a Regression Test Repository (called: the Repo) are to:

• Explain the purpose of building up the Repo,
• Mention the activities needed to buildup and maintain the Repo,
• Enlist some types of regression test execution regarding Repo,
• Highlight some of the assumptions,
• Provide a set of the benefits of using this kind of Repo maintenance style.

The purpose of the Test Repo buildup

Is to be able to quickly adapt Test Case Repository by implementing a simple yet efficient method of incorporating Test Case changes made in Sprint Testing (see blogpost 1). By having this method implemented, we can be more confident in Regression Test Coverage and even know the current state of the implemented solution (given we are using a Behaviour-Driven Development style of Test Case Design).

The Regression Test Repo buildup activities

Each part of the application from the part 1 of this blogpost series can be represented in the Repo by the application path. The application path has a following format:

<Flow>.<Service Scope>.<Page>.<Sub-page> →  Sales.Gateway&Acquiring.Gateway services.Technical setup

Buildup activities are required to be done on each part of the application.

An explanation of the regression test suite buildup process per each part follows below.

1. Initial buildup of the Regression Test Repo

This process is needed only if the Regression Test Repo has not been built yet. In our case, we had a lot of older sprint test cases that we consider to be an initial source Test Suites that were to be analysed and brought into the initial Test Repo.

This process consists of few steps:

• Analyse the source Test Suites to identify the relevant tests that should be included in the regression suite,
• Add the relevant tests from the source test suites into the regression suite AND cleanup the obvious stuff like recent changes in terminology (example: renaming 'Face to face' into 'Card present' and 'Ecommerce' to 'Card not present' when talking about payment transaction type names),
• Review and refine the Regression Test Repo and switch to BDD style of test cases.

2. Incorporating Sprint Test Cases into the Regression Test Suite

This process is done after the initial buildup of Regression Test Repo, or right from the start, periodically, if you have empty Test Repository. The suggestion is to start from the Sprint Test Repo and bring in any Test Cases from there to gradually buildup the Repo.

The steps are:

• Analyse the relevant sprint test cases and see if they are candidates for Regression Test Case Repo
• Add new/Change/Delete test cases from the sprint test cases suite to the regression suite (explained in more details below in the section: Guidelines for incorporating sprint test cases)
• Review and refine the regression suite as necessary to ensure complete coverage.
• Cleanup source suites from step 1 - 1. Initial buildup of the Regression Test Repo if necessary

Notes:

• The execution of the Regression test suite is not included in these steps,
• The person building up the regression repo should enlist all the nuances in this process and adjust the updating procedure to ensure seamless maintainability of the Regression suite.

Guidelines for incorporating sprint test cases

While incorporating sprint test cases, one should consider three ways of doing that task:

• Adding a brand-new test case: for a newly introduced functionality. For example: adding a new sub-page to an existing page group, for example: Test Cases for Navigation Tree, Application State Transitions, and the new sub-page related Test cases,
• Changing or adding Description, Precondition, Expected Result or Test Step: for existing Test Cases,
• Removal of the Test Case: for removed functionalities.

Execution and Reporting notes

Frequency of execution

The execution of Regression Test Cases in our case is done after thoughtful consideration of the effects of current Release issues on the application. Since the Regression Test Repository is organized by the application elements, it is very intuitive to incorporate and later do a regression test of specific parts of the application once, before the Release is done.

If sprint Test Cases incorporation is taken care of periodically and diligently, i.e. once every two sprints, then the QA team always has a point of reference and can pick a set of tests to reliably check for Regression bugs.

Types of execution

We can mostly use the Regression Test Plan, that is Full in its structure, if we want to do a full regression test. Then, we can test Partially – which is mostly done while testing before the Release – sometimes, even in combination with parts of the Sprint Test Repository if the changes have not been incorporated. Finally, the Test Plan can be Clean if we are using only the Regression Test Repository.

The figure below depicts the types of execution we can have:

Given the real-life Frequency and Type of regression chosen, one example of the Regression Test Plan can be seen on the figure below:

Assumptions and limitations

There are also a few remarks about assumptions for using this style of Test Repository maintenance:

• Team Structure: Assumes a team structure of at least one QA per two DEVs,
• Clean Requirements: Requires clean requirements in Behaviour-Driven Development (BDD) style,
• Development Methodology: We assume that the team uses the Scrum methodology,
• Applicability: While the method has not been tried in other project setups, there is a possibility to port these tactics to other types of Test Case Design, aided by OpenAI API, for example. Caution is advised, and that process should be approached with due diligence.

Benefits of using this Repo maintenance style

Preparation for automation is a key benefit of the described test repository approach, because it offers us:

• insightful overview and quality control of product development over time,
• tracking how changes affect regression errors on various parts of the application,
• a valuable reference point for designing automated regression tests in tools like Cypress or SpecFlow. In this way, this style of the Repo maintenance could serve as a basis for such kind of automation.

Conclusion

In this second part of blog series, the focus has shifted to a vital aspect of maintaining the Test Case Repository—specifically, the incorporation of initial repo and of Sprint Test Case repo changes into the Regression Test Repo.

The aim of this blog series was to highlight how to quickly adapt the Test Case Repository, simplifying the integration of changes from Sprint Testing, thereby enhancing confidence in Regression Test Coverage, particularly when utilizing Behaviour Driven Development (BDD) for Test Case Design. To build the repository, each part of the application is carefully represented, then we incorporate initial setup and gradually integrate new Sprint Test Cases to improve coverage and refine the suite for better testing outcomes.

We have also mentioned how the execution of the Repo test cases is done by evaluating current Release issues, providing flexibility with dividing execution types by Scope and by Structure as depicted here - Figure 1: Types of regression test execution.

Key assumptions & limitations, as well as benefits of using this type of test Repo maintenance are briefly mentioned.

Finally, this blog series (see: first part) provides a guide to building and maintaining a Regression Test Repository. We emphasized the purpose, necessary activities and offered practical examples and guidelines for implementation of such kind of repo maintenance style.