Nowadays test automation is a must, yet lots of companies are struggling with automation. Test automation is especially difficult during short sprints. According to a recent Capgemini case study, only 22% of the test cases are automated during sprints. The cause is that test automation starts when the implementation is ready and test automation is a time-consuming task.

We also faced the problem of slow test automation. Finally, we realized that we should do test automation in two steps:

1. First, we should do the implementation-independent part of test automation that is test design. We use a new method called action-state testing. Here abstract test steps are created resulting in a model. From the model abstract tests are generated that can be executed by the tester, but not a machine. This should happen before the implementation.

2. Creating the test cases based on the action-state model. The model guides the testers to create executable test cases in a very short time.

Action-state testing is introduced in the book Paradigm Shift in Software Testing will be published soon. This is a step-by-step test design technique where abstract test steps are created one by one from a given state, supported by an algorithm. Instead of making a whole model, it is made gradually from a starting point that is the initial state. In our agile world, making a whole model is usually not possible as the features are selected and implemented gradually. Action-state testing conforms with this concept. In action-state testing, we start from an initial state and investigate which actions can be done. Then we select an action such as ‘add a coke to the cart’ and investigate the system’s response and the arriving state when this action has been finished. The response can be ‘two cokes in the cart’, the state can be ‘total cost is below to pay’. According to this our first abstract test step is: (‘add coke to the cart’, ‘two cokes in the cart’, ‘total cost is below to pay’). Then we add another step from the initial state or from the state ‘total cost is below to pay’.

Action-state testing is an implementation-independent test first method. You can do it at the beginning of each sprint. This is the larger part of the whole test design + automation process, thus the test cases can be automated in-sprint.

In an abstract step, the user actions are the inputs, responses are the outputs to be validated and the states are the cornerstones showing the subsequent steps. The only requirement for such an abstract test step is that the tester be able to execute the step and validate the result. Our step: (‘add coke to the cart’, ‘two cokes in the cart’, ‘total cost is below to pay’) is an appropriate step as the tester can add a coke, validate the cart to contains two cokes independently from any implementation (if not, then this is an issue to be fixed).

In action-state testing, we build a model by adding abstract test steps. We always start from a state and investigate the possible actions from here. By doing the action, the systems responses can be validated. After the system responses, it arrives at a state that can be the same as we started. Creating abstract test steps, we also build a state transition graph. This graph helps you to know which are the missing actions if any. If we start from an initial state

INITIAL STATE state 0

and the first step is

first action => first response STATE state 1

then in the background, we get the following state transition diagram:

You can display the current graph after adding a new step. The graph can be used to realize any missing action or state. For example, if there is an action from state 1 that arrives also state 1, you immediately realize that this action is missing. Adding the following step:

second action => second response STATE state 1

the graph contains an additional edge labeled by ‘second action’:

You should add new steps until the state transition graph contains all the available actions and test states. This is the first phase of action-state testing. Note that the first phase contains all the basic tests that cannot be ignored in any case. This is the minimum requirement for testing any application.

When the first phase is ready, you can start the second. In this phase, Harmony offers the necessary steps to satisfy the build-in test selection criterion. Currently, one test selection criterion is in Harmony that is the all-transition-pair criterion that requires that each adjacent transition pair be covered.

Harmony marks the steps that may be extended by child steps. For example, the well-known ATM authentication feature has the following model after the first phase:

The marked steps may have child steps. You should investigate each offered step and either accept or reject it. When all the steps for which Harmony offered child steps are ready (✓), you are ready. This means that the generated test cases satisfy the all-transition pairs test selection criterion. As a consequence, the tests will detect lots of tricky bugs.

It’s very important that making the first step is a must, but making the second step is very fast. It takes much less time than fixing the tricky bugs some months later. Completing the second phase doesn’t require any step creation or test execution automation only checking the feasibility. Therefore, we strongly suggest doing it.

How to make locators stable

We know for more than 20 years that the capture and replay method fails. Even for a seemingly small change, the locators should also be changed, and the test case becomes obsolete. That’s why test automation engineers are looking for stable locators for GUI objects. In general, you can always use XPath expressions, while using better options, such as CSS selectors, ID or name are accidental. For those who are not familiar with this field, ‘XPath (XML Path Language) is a query language for selecting nodes from an XML document.’ Every UI object to which we assign a value or validate a result must be localized. XPath expression is in fact a path containing nodes on that path. This is very much like the path expressions you use with traditional computer file systems.

Now let’s consider XPath expressions. Most test automation engineers complain about the maintainability of their test code. If the code modifies, then XPath expression should also be modified.

In Harmony, we introduced an applicable and simple solution. Let’s forget XPath and use a special attribute called ‘data-harmony-id’ instead. This should be coded, however, it’s much faster than creating a stable XPath expression. A simple example is data-harmony-id=’Login button’. In this case, the related category declaration is:

Login button(A): #pressed

In practice, there are more difficult cases. Let us assume, for example, that we have more projects and for each project, the project owner can assign different contributors. How can we create a good id for similar delete buttons? It’s reasonable to create understandable and meaningful IDs. In our example, the ids of deletion of Contributor3 and Contributor4 in project ‘First’ can be:

‘In project First delete icon of Contributor3@example.com’
‘In project First delete icon of Contributor4@example.com’

To do this the developer should include the following code:

With this, very elegant test cases can be designed for example by applying an extended Gherkin syntax:

WHEN In project First delete icon of User3@example.com IS #pressed

THEN …

Cool, isn’t it? The developer extends the code with a single element. Here is an example in JavaScript with React):

<td  data-harmony-id={`In project ${title} project member ${member.name}`}>

Coding an XNode takes less than half a minute for a developer while finding an appropriate XPath expression takes about five minutes for a test automation engineer on average. You can consider this as an additional requirement by which the software quality is improved a lot. In this way the locators become predefined and stable, i.e., you can modify GUI in any way, the test case will pass. You should change the test only if the requirements are changed.

Most of the current codeless test automation tools ‘help’ testers with scanning the screen and making appropriate initial XPath expressions. It helps for the first time, however, as the code changes, the related test cases become outdated and should be continuously fixed. In many companies, test automation engineers should deal with test code maintenance resulting in the lack of newly automated test cases.

To summarizing, stable codeless test automation code can be created by effective teamwork of developers and testers. Developers can use understandable test cases for implementation resulting in a higher quality code. Maintenance becomes simple and cheap. And it’s not just a theory.

How to avoid maintenance problems in test automation

One of the main problems concerning test automation is maintenance. As we have more and more test cases they require huge rework. There are two reasons for this. The first is that test cases are cloned many times and during maintenance, all of them should be modified. The second is that code is changing frequently and the locators of GUI objects should also be modified frequently. Therefore, in the beginning, the test automation engineers can make lots of automated test cases, but fewer and fewer as time goes by. The result is the following figure:

This means that considering a given amount of resources, i.e. testers, the number of test cases will not increase after some time and the test engineers will only maintain their test code.

To avoid test code duplication, we introduced use case test design into the test automation process. In this way, test cases are created in a top-down way and no duplication occurs. On the other hand, instead of coded test cases, we apply a higher level by extended the Gherkin language that is used by developers during BDD (Behavior Driven Development). Gherkin is a domain-specific language. It’s a test first method, where test cases are implementation-independent. Gherkin helps to describe business behavior and acts as documentation and skeleton of your automated tests. Here is a simple example:

Feature: Login
  Scenario: Happy path
    When I insert my account
    And I insert my password
    Then I’m logged in
  Scenario: Wrong password
    When I insert my account
    And I insert a wrong password
    Then a message “wrong login name or password” appears 

In Gherkin the test code, i.e. “step definition” is written based on the Given-When-Then steps:

public class MyStepdefs {

@When(“I insert my account”)

public void …

This means that BDD needs coding of the test cases. This is the transition from implementation independence to implementation dependence. Besides this is a huge work, and the maintenance is also problematic. When a requirement has been changed you should modify the Gherkin steps and then the code. However, usually, only the code is modified, therefore development pipeline requirement – implementation-independent test cases – test code – application code will not be consistent.

In Harmony, the original Gherkin has been extended to Gherkin++. Instead of scenarios, you can make acceptance criteria. An acceptance criterion is a part of a test case that also contains pre and postconditions. Let’s consider an example.

Requirement. If the login is successful, then a message appears ‘Login successful!‘

This can be modeled by a single action-state step:

The first user enters correct login data => system accept login data  STATE first user logged in

From this the tester makes executable test steps:

The first user enters correct login data:
WHEN Login name IS Smith
AND Password IS 2a4b6c
WHEN Next IS #pressed

system accepts login data:
THEN Msg IS Login successful!

Here the action and the response are extended to two ‘clauses’ that are executable test steps. The green words are the keywords. The blue words are the categories, which are the parameters and the white text are the choices, i.e., the values. The specific character ‘#’ helps the test code generator to create keywords such as ClickOn. This clause has to be extended by a precondition, which starts the application and selects the feature Login.  The generated Java code is as follows:

EN.SetValue(“URL”, “https://cloud.4test.io/pizza/”);
EN.SelectWindow(“Main menu”);
EN.ClickOn(“Log in”);
EN.SelectWindow(“Login”);
EN.SetValue(“Login name”, “Smith”);
EN.SetValue(“Password”, “2a4b6c”);
EN.ClickOn(“Next”);
EN.VerifyValue(“Msg”, “Login successful!”);
EN.ClickOn(“Next”);
EN.SelectWindow(“Main menu”);
EN.ClickOn(“Exit”);

The result is that you never write or modify any coded test case. You should only modify the model and/or the clauses.

To address the locator problems read the previous article.