Harmony documentation

Harmony’s first use quick guide.

Download

White papers

Why test design is possible at all?

Every tester knows that lots of bugs can be found by applying appropriate test design techniques, though the number of test cases is negligible comparing all the possible test cases, i.e. the whole input domain. But why? The reason is the competent programmer hypothesis i.e. “Programmers have one great advantage that is almost never exploited: they create programs that are close to being correct”.

Why test design is possible at all?

Every tester knows that lots of bugs can be found by applying appropriate test design techniques, though the number of test cases is negligible comparing all the possible test cases, i.e. the whole input domain. But why? Dear reader, we think that you should know why it is possible. The reason is the competent programmer hypothesis (CPH), which was identified by DeMillo et al. in 1978.  They observed that “Programmers have one great advantage that is almost never exploited: they create programs that are close to being correct”. Developers do not implement software randomly. They start from a specification, and the software will be very similar to their expectations, hence, close to the specification.

CPH makes test design possible. If the implemented code could be anything then we should test the application to differentiate it from an infinitive number of alternatives. This would need an infinitive number of test cases. Fortunately, based on CPH, we only have to test the application to separate it from the alternative specifications which are very close to the one being implemented. In order to demonstrate CPH, let’s consider an example:

R1. Input two integers, say x and y from the standard input device.
R2. If x and y are both negative, then print x + y to the standard output device.
R3. If x and y are both positive, then print their greatest common divisor to the standard output device.
R4. If x and y have opposite signs (one of them is positive and the other is negative), then print their product to the standard output device.
R5. If x·y is zero then print zero to the standard output device.

The test set T = { ([-2, -3]; -5), ([4, 2]; 2), ([2, -4]; -8), ([2, 0]; 0) } is adequate, it achieves 100% requirements coverage. Consider an alternative specification:

R1. Input two integers, say x and y from the standard input device.
R2. If x and y are both negative, then print x + y to the standard output device.
R3’. If x and y are both positive and |x – y|<10, then print their greatest common divisor otherwise the smaller value to the standard output device.
R4. If x and y have opposite signs (one of them is positive and the other is negative), then print their product to the standard output device.
R5. If x·y is zero then print zero to the standard output device.

Assume that the developer implemented this specification. In this case, T would not be adequate even if all test cases would pass. What’s more, any positive pair with  a difference less than 10 would not be reliable. Fortunately, because of the competent programmer hypothesis, it is unrealistic that the programmer implements R3’, since this requirement is far from the original.

However, the inverse case may happen with a much higher probability, i.e. R3’ is the part of the specification and R3 will be implemented. Fortunately, in this case the original T is not reliable as the tester should design test cases for |x-y| = 9 and |x-y| = 10, so that either x or y is greater. In this way the bug will be detected.

Based on CPH we can measure the quality of a test set. The method is called mutation testing. The idea is to slightly modify the specified code in many code location. For example instead of if (x > 2), the mutant code contains if (x >= 2). This is a small change and even competent developer can do it. If the test set ‘kills’ all the mutants, then it’s  reliable test set. Killing a mutant means that there is one test case for which the correct code passes, but the mutant fails. If you want to test your test design experience let’s try to test some examples here.  These examples test your knowledge by applying mutant code.

If you are interested this topic, read this book!

Test design should also be automated

Nowadays test automation is a must, yet lots of companies are struggling with automation. Many people fall into the trap that they do not know what to automate: not the test cases should be automated but the test design. Harmony is a perfect tool for automate test design in a test first manner. We apply a Gherkin++ language, which is easy to learn and use.

Test design should also be automated – use case testing

Nowadays test automation is a must, yet lots of companies are struggling with automation. Test automation is especiallydifficult during short sprints. According to a recent Capgemini case study, only 22% of the test cases are automated during sprints. The cause is that test automation starts when the implementation is ready but traditional test automation is a difficult and time-consuming task.

Slow test automation is the bottleneck of the development. We have elaborated an appropriate process to overcome this problem. We realized that the solution of test automation shall be done  in two steps:

  1. First, we should do the implementation-independent part of test automation that is test design. The best solution is use case testing as it’s simple and efficient. From use cases, manually executed abstract tests can be generated. This should happen before the implementation.
  2. Creating the test cases based on the use cases. Use case steps lead us to create executable test cases in a very short time.

Use cases were introduced in the late eighties by I. Jacobson and started using for object-oriented software engineering. A use case is an appropriate description of how the user wants to use an application. A use case captures all the possible ways the user and system should interact to achieve their goals. The original concept was extended by A. Cockburn.

Use cases provide a formal way to write the requirements from the perspective of the end-user. An appropriate use case representation shows the developer what to code, the customer what to expect, and the tester what to test.

The testing community realized that the use case model can be applied as a test design technique. The technique is simple, just consider each scenario obtained from the use case as an abstract test case. Nowadays use case testing is the most popular test design technique used by 73% of the testers worldwide.

Use cases consist of (use case) steps. There are two types of steps:

  1. User action – an action the user can do in the system. It’s an input or a set of inputs such as inserting a password.
  2. System validation – for a user action the system does what it should do. For example, entering a password the system checks the password and displays a related message. A validation step is simple to check the required output.

Applying use case testing makes it possible to automate test cases very fast, even in every sprint. Use cases are implementation-independent and should be created when the requirements are ready or at the beginning of a sprint. Use cases are the high-level description of the processes in an application, therefore it’s much easier to develop than test cases with concrete values. For example, ‘add articles to the shopping cart to make ordering possible’ is understandable for everyone and a manual tester can do it.  This cannot be directly automated, but based on this step, automation becomes easy. For example, the concrete test description can be the following:

WHEN menu Shopping IS #pressed
WHEN Add beer icon IS #pressed
WHEN Add pizza chicken IS #pressed

In this way, we split our work into two parts and the complexity for one task became much less. Our experience is that test automation time reduced to 30-50% of the original.

A use case step has a level. The first action has the highest level. If action B can happen only after action A having a level k, then the level of B is k-1. Creating use cases, we use simple numbering. Each use case step id starts with ‘@’. At the first level, we label with numbers, 1, 2, etc., at the second level with letters, a, b, c, etc., at the third level with numbers again, etc. For example, if action B should happen after action A at level 1a then we write

@1a action A
@1a1 action B

If action A and action B can happen in any order, i.e. action A can follow action B and vice versa, then the number is the following:

@1a action A
@1b action B

In the first case, Harmony generates one test, in the second case two test cases Here is an example of a use case and the generated abstract test cases:

The grey lines are the name of the test cases that are the last use case steps.

If you want to continue a test case with the subsequent use case steps, you can use an ellipsis, i.e. …

Here is an example:

When you are ready with the use cases, your team should validate the generated abstract tests. This is a very good defect prevention possibility. Abstract tests are also a good starting point of the development work. Case studies proved that relying on TDD, developers inserted fewer bugs into the code. Abstract tests are better than TDD as they become understandable.

Can State transition testing avoid combinatorial explosion?

Most applications are stateful. State transition testing tries to find bugs concerning erroneous transitions resulting in faulty states. The key is a good test selection criterion. We show a complex criterion by which tricky, higher-order bugs can be detected. We also show that applying this criterion leads to a double number of test cases than a simple criterion.

Register to receive regular white papers

Combinative testing – find tricky bugs with a linear number of test cases

Combinatorial testing aims to determine data combinations to be tested. In some cases, when the risk is high or the application is very complex, you probably use combinatorial testing. The linear and practically usable alternative of combinatorial testing is diff-pair testing. Informally, it requires that each value p of any parameter P be tested with at least two different values q and r for any other parameters. It’s linear and easy to use.

Register to receive regular white papers

EP+BVA is not so simple as you have learned

Equivalence partitioning (EP) and boundary value analysis (BVA) together are widely used techniques. It’s easy to understand and used. The test selection criterion of boundary value analysis requires testing an ON, an OFF, and an IN/OUT point for each border.  The ON/OFF point is a point on the closed/open border. They are as close to each other as possible. The IN/OUT point is a point at the same EP as the ON/OFF point.

Register to receive regular white papers

Test design

Test design is not a hot topic. However, it should be as the design test cases determines the quality of the application. The surprising fact is that applying appropriate test design techniques, by which you are capable of finding more bugs, requires only a little more expensive testing. What is more, the total SDLC costs are decreased.  How is it possible? Read more.

Why risk analysis is really necessary for test design?

The cost of projects is strongly influenced by two factors. One is the cost of testing (designing and executing the tests, building the testing environment, etc.), the other is the cost of defect correction (through the SDLC). Fortunately, the total cost has an optimum value determining the required testing effort. Thus the test design techniques should be selected based on the risks and the complexity.

Register to receive regular white papers

How to make locators stable

Test cases and code are connected via locators. Unfortunately, code modification results in the test cases become obsolete. In Harmony, we introduced an applicable and simple solution. Let’s use a special attribute called ‘data-harmony-id’ instead. This should be coded, however, it’s much faster than creating a stable XPath expression.

How to make locators stable

We know for more than 20 years that the capture and replay method fails. Even for a seemingly small change, the locators should also be changed, and the test case becomes obsolete. That’s why test automation engineers are looking for stable locators for GUI objects. In general, you can always use XPath expressions, while using better options, such as CSS selectors, ID or name are accidental. For those who are not familiar with this field, ‘XPath (XML Path Language) is a query language for selecting nodes from an XML document.’ Every UI object to which we assign a value or validate a result must be localized. XPath expression is in fact a path containing nodes on that path. This is very much like the path expressions you use with traditional computer file systems.

Now let’s consider XPath expressions. Most test automation engineers complain about the maintainability of their test code. If the code modifies, then XPath expression should also be modified.

In Harmony we introduced an applicable and simple solution. Let’s forget XPath and use a special attribute called ‘data-harmony-id’ instead. This should be coded, however, it’s much faster than creating stable XPath expression. A simple example is data-harmony-id=’Login button’. In this case, the related category declaration is:

Login button(A): #pressed

In practice there are more difficult cases. Let us assume, for example, that we have more projects and for each project the project owner can assign different contributors. How we can create a good id for the similar delete buttons. It’s reasonable to create understandable and meaningful id’s In our example, the ids of deletion of Contributor3 and Contributor4 in project ‘First’ can be:

‘In project First delete icon of Contributor3@example.com’
‘In project First delete icon of Contributor4@example.com’

To do this the developer should include the following code:

With this, very elegant test cases can be designed for example by applying an extended Gherkin syntax:

WHEN In project First delete icon of User3@example.com IS #pressed

THEN …

Cool, isn’t it? The developer extends the code with a single element. Here is an example in JavaScript with React):

<td data-harmony-id={`In project ${title} project member ${member.name}`}>

Coding an XNode takes less than half a minute for a developer while finding an appropriate XPath expression takes about five minutes for a test automation engineer on average. You can consider this as an additional requirement by which the software quality is improved a lot. In this way the locators become predefined and stable, i.e., you can modify GUI in any way, the test case will pass. You should change the test only if the requirements are changed.

Most of the current codeless test automation tools ‘help’ testers with scanning the screen and making appropriate initial XPath expressions. It helps for the first time, however, as the code changes, the related test cases become outdated and should be continuously fixed. In many companies, test automation engineers should deal with test code maintenance resulting in the lack of newly automated test cases.

To summarizing, stable codeless test automation code can be created by effective teamwork of developers and testers. Developers can use understandable test cases for implementation resulting in a higher quality code. Maintenance becomes simple and cheap. And it’s not just a theory.

How to avoid maintenance problems in test automation

Test cases are cloned many times and during maintenance, and all of them should be modified. To avoid test code duplication, we introduced use case test design into the test automation process. In this way, test cases are created in a top-down way and no duplication occurs. We also apply an extended Gherkin language called Gherkin++.

How to avoid maintenance problems in test automation

One of the main problems concerning test automation is maintenance. As we have more and more test cases they require huge rework. There are two reasons for this. The first is that test cases are cloned many times and during maintenance all of them should be modified. The second is that code is changing frequently and the locators of GUI objects should also be modified frequently. Therefore, in the beginning the test automation engineers can make lots of automated test cases, but fewer and fewer as time goes by. The result is the following figure:

This means that considering a given amount of resources, i.e. testers, the number of test cases will not increase after some time and the test engineers will only maintain their test code.

To avoid test code duplication, we introduced use case test design into the test automation process. In this way test cases are created in a top-down way and no duplication occurs. On the other hands instead of coded test cases we apply a higher level by extended the Gherkin language that is used by developers during BDD (Behavior Driven Development). Gherkin is a domain specific language. It’s a test first method, where test cases are implementation-independent. Gherkin helps to describe business behavior and acts as documentation and skeleton of your automated tests. Here is a simple example:

Feature: Login
  Scenario: Happy path
    When I insert my account
    And I insert my password
    Then I’m logged in
  Scenario: Wrong password
    When I insert my account
    And I insert a wrong password
    Then a message “wrong login name or password” appears 

In Gherkin the test code, i.e. “step definition” is written based on the Given-When-Then steps:

public class MyStepdefs {

@When(“I insert my account”)

public void …

This means that BDD needs coding of the test cases. This is the transition from implementation-independence to implementation-dependence. Besides this is a huge work, and the maintenance is also problematic. When a requirement has been changed you should modify the Gherkin steps and then the code. However, usually, only the code is modified, therefore development pipeline requirement – implementation-independent test cases – test code – application code will not be consistent.

In Harmony, the original Gherkin has been extended to Gherkin++. Instead of scenarios you can make acceptance criteria. An acceptance criterion is a part of a test case which also contains pre and post conditions. Let’s consider an example.

Requirement. If the login is successful, then a message appears ‘Login successful!

The related acceptance criterion (AC) is:

Successful:
WHEN Login name IS Smith
AND Password IS 2a4b6c
WHEN Next IS #pressed
THEN Msg IS Login successful!

where Successful is the name of the AC and the generated test case as well. The green words are the keywords. The blue words are the categories, which are the parameters and the black text are the choices, i.e. the values. The specific character ‘#’ helps the test code generator to create keyword such as ClickOn. This AC has to be extended by a precondition, which starts the application and selects the feature Login.  The generated Java code is as follows:

EN.SetValue(“URL”, “https://cloud.4test.io/pizza/”);
EN.SelectWindow(“Main menu”);
EN.ClickOn(“Log in”);
EN.SelectWindow(“Login”);
EN.SetValue(“Login name”, “Smith”);
EN.SetValue(“Password”, “2a4b6c”);
EN.ClickOn(“Next”);
EN.VerifyValue(“Msg”, “Login successful!”);
EN.ClickOn(“Next”);
EN.SelectWindow(“Main menu”);
EN.ClickOn(“Exit”);

The result is that you never write or modify any coded test case. You should only modify the use cases and the acceptance criteria, in many cases only the second. However, you should only modify the acceptance criteria that are related to a modified requirement.

Register to receive regular white papers