Creating of an efficient Test Case repository – Part 1

Welcome to our proposed two-part blog post series on creating an efficient Test Case repository. We initiated this series to address issues in our current repository setup, specifically regarding Test Case creation and execution. Our aim was to optimize QA processes by reorganizing the repository and implementing BDD principles via Gherkin scenarios.

The redesigned repository aimed to enhance Test Case creation and execution, concentrating on eliminating redundancy and emphasizing precise test purposes. This Test Case Design approach resulted in concise and effective test steps.

This blog post series will consist of two parts. The first part delves into comparing our initial and target Test Case Repository structure, offering insights into the repository hierarchical re-organization and Test Case design/creation changes and explains how Sprint Test Cases are executed and reported on.

The second part will bring focus to organizational procedures needed for maintaining a Test Case Repository by explaining the process of incorporation of Sprint Test Case changes into the Regression Test Suite – in other words, we talk about Regression Test Suite maintenance.

By following these two blog posts, you'll learn how to create test cases aligned with each sprint's user story, confidently approve sprint-induced changes, manage test reporting, and build a process to enable Test Case reusability while building the Regression Test Suite. The final product of these activities will be a well-organized Test Repository. The repository built in such a way can even serve as a foundation for BDD scenario driven test automation utilizing some of the popular tools like Selenium (SpecFlow) or Cypress.

The goals of the first blog post are to show how:

•  we introduced changes into the existing Test Case repository hierarchical structure in TestRail and optimized it for our testing team

• we kept our Sprint Test Cases simple and easy to create, run and report on their execution.

The path to the Target Test Case repository structure

The Initial and Target repository will be depicted on a somewhat imaginary project structure to keep sensitive data safe. After a brief intro to each of the structures, they will be compared side by side.

Initial Test Repository structure

The Test Case repository was organized in Test Rail, with the Multiple Test Suite project type. Each User Story was represented by one Test Suite and the Regression Test Case repository did not exist. In the initial phases of the project, Test Suites were a direct reference to the Epic, with each Epic covering a larger set of logically linked User Stories as is most often the case when working with Scrum methodology. By following that momentum, the user stories were started to be wrongly added as new standalone Test Suites and it would soon get crowdy.

Figure 1 (above) shows that there is no Sprint reference whatsoever. Additionally, the figure shows 7 out of 16 User Stories and it is already getting crowded in the Dashboard page.  Furthermore, there is not even a mention of the Regression or Sprint Test Cases here. All of these conditions meant that the Test Case creation and organization, as well as execution and reporting could soon get pretty messy if not taken care of systematically.

The next chapter goes into detail how we have decided to set up the Test Repository.

Target repository structure

Test Suite redesign was tackled with several key concepts we have agreed to apply:

• The project type in TestRail should be a Single suite repository instead of a multiple suite repository to ensure top-level readability and avoid issues in finding specific Test Cases as noticed in the initial Test Repository setup.
• The Sprint Test Cases and Regression Test Cases should be separated into two top-level Test Rail Sections (in further text we will call them folders):
  • Regression Test Cases – should firstly be functionally organized
  • Sprint Test Cases – should be organized in TestRail folders that clearly indicate the exact Sprint and User Story details. The template for quick generation of User Story test cases was to be introduced to kick-start Test Cased Design during the sprint
• The Test Cases should be written in a Behavior Driven Development style, by using Gherkin syntax
• The existing/initial Test Cases should be filtered out, ported to the Regression Test Repository and then translated to the Gherkin syntax 
• The Sprint Test Cases should be periodically incorporated into the Regression Test Repository - this process we called Regression Suite Buildup

Given the concepts presented above, we decided to have Multiple Test Suite repository ported to a single repository by a series of steps:

1. Create Regression and Sprint folders by following the key concepts mentioned above
2. Test Case Redesign - detail the BDD style by using Gherkin syntax
3. Start using Sprint and User Story folders for Test Case creation and execution
4. Gradually build up the Regression Test Repository out of existing Test Cases
5. Implement procedure for incorporating Sprint Test Cases into the Regression Test Repository
6. Apply the procedure for incorporating Sprint Test Cases into the Regression Test repository

1.      Create Regression and Sprint TestRail folders

Regression Test folder

Regression Test Section consists of several layers deep test folders based mostly on process and functional division. Therefore, since the software under test has a process flow approach, each stage in the process is called a flow, and it is represented by its own folder. To name a few of them, we have: Sales flow, Merchant flow, Underwriting flow, Onboarding flow…

Each of the functional areas contains page level test folders, such as: Sales flow.Business Details, Merchant flow.Business details, Sales flow.Gateway services.Technical setup. Finally, under the page level, there are Test Cases and test case naming is done in the following format:

{null  (in case of modal page or single section page) | page section (if exists)} | {action} and/or {expected result}. For example:

• Test Case Title on the page with multiple sections: {null} | Integrated terminals | The user is shown an error if they don’t add a terminal and try to continue
• Test Case Title on the page with only one section: The user can see a list of company types

Besides functionally organized folders, there are also folders that cover Application Status Transitions, Integrations, Generic and Admin.

The example of a page level folder and a few Test Case Titles is on the Figure 2 (below):

Figure 3 (below) shows one way of how you can organize the structure of the Regression Tests folder. Test Case titles/names have already been dealt with, so I will leave the exact titles to your judgment.

Sprint Test Section/folder

Each Sprint folder contains a short Sprint description and link to external project in JIRA. Under the Sprint folder, there are User Story sub-folders with their own template (User Story Link, Epic Link, short description of the user story). Finally, on the lowest level there are Test Cases.

The sample structure of the Sprint Test folder can be found on the Figure 5 (below).

Test cases are named in the following format:

[{user/functional area}]  {test target element} | {test description}

Few Sprint Test Case name examples are found below:

• [Sales] Card present | Integrated terminals | The user can verify changes as they were entered by the Admin User
• [Admin] User Management | The user can add a new user to the existing role

Importantly enough, in the Sprint Test Case title we define information about the user/functional area because we don’t have a functional division in Sprint Test Repository like it is done in the Regression Test Section/folder. Furthermore, these prefixes are a good indication for deciding where we should possibly update/delete/add Regression Test Case/s in the Regression Test Repository.

The example of a user story implementation can be found in 5 (below).

Sprint and User story template folder

For speeding up the Sprint Test Case Design process, we prepared the template Sprint and User Story folder which is to be instantiated for each sprint and each user story.

The templates look like as can be seen in Figure 5 and Figure 6, respectively.

As seen in the figure above, the language tests, data migration tests and user role tests are defined as a part of the User Story template as well. Therefore, each User Story template has a set of predefined test cases which cover these topics. When needed, these tests can be further detailed, and when not needed, they can be easily removed from the instantiated folder for the current User Story.

2.     Test Case Redesign - BDD style by using Gherkin syntax

Test case redesign - focuses on enabling other fields in the test case to become more relevant by using Description field filled in BDD style. As a consequence, this enabled us to map certain parts of the BDD based description to Preconditions (GIVEN clause), Expected results (THEN clause) and concise Test Steps (WHEN clause). The way of defining BDD test cases very closely follows instructions for Gherkin language described on the SpecFlow’s website here (if you want more general introduction into the Gherkin language) and here (if you already know about Gherkin, but you are not sure what goes into Given and When clauses). One simple Gherkin test scenario is displayed below:

Feature: Calculator

Calculator for adding two numbers

Scenario: Add two numbers

Add two numbers with the calculator

Given I have entered 50 into the calculator

And I have entered 70 into the calculator

When I press add

Then the result should be 120 on the screen

Following these practices enabled a straightforward and simple higher order test cases which contain abstract description of certain actions and effects (not focusing on expressions like ‘click on the button Next on the footer’, but rather on ‘the user advances to the next page’. Written this way, Test Cases are more robust and maintainable. They are essentially encapsulating exact actions in a form of Gherkin expressions.
This kind of Test Case Design is a good preparation for Test Automation Experts, and they behave as a template for Gherkin scenarios-based Test Automation tool/s such as SpecFlow or Cypress with Cucumber.

The comparison of the new and old style can be found on the Figure 7 (below).

Clearly, there is an immediate difference between the New and Old (right) style. Namely, the old style has 4 more steps noted down below, without adding any meaningful context to the tester because the first two steps are just there for Login actions. At the same time, the new Test Case style displays all of the checks needed to be done on the page in just one step. All the preconditions are named clearly and they follow the Test Description which is simply written in the BDD style. Everything fits together like a glove.

3.     Start using Sprint and User Story folders for Test Case creation and execution

Following the Sprint structure explained above (Sprint Test Section/folder), we can naturally copy/paste predefined Template folders for Sprint and User Stories, and just change a few important pieces of information, like JIRA issue id’s of Epic, User Story etc.

Furthermore, it is necessary to follow the new style of writing Test Cases.

Test Run Creation and Execution

With these two prerequisites, you only need to create a new Test Run or Test Plan in TestRail, and bring in the desired folders.

The benefit of using one Sprints folder is that you can easily bring in any sub-folder into the current sprint’s Test Run. Moreover, if you want, you can bring in the test cases from previous sprints in the case of carrying over User Stories to the next sprint. In this case, you will always have a reference which test cases were initially part of another sprint, and you will know which ones are a part of the current sprint.

Test Plan prepared in such a way can be seen below in Figure 8: Test Plan - executed.

When needed, we can also use Milestones and further group Test Runs or Test Plans, depending on the project structure and needs for reporting.

Reporting

Finally, reporting comes naturally out of all the Sprint testing steps mentioned in the previous sections.

One report example can be found below in Figure 9: Test Report.

Thank you for joining us on this exploration of Test Case repository refinement. Stay tuned for more insights and actionable strategies in our forthcoming discussions.