Jovica Krstevski

Technical Lead

Aleksandra Angelovska

Quality Assurance Lead

In previous articles, we discussed “E2E development challenges and overview of the E2E testing frameworks” and “E2E Test Components”. The goal of this article is to simulate what a real user scenario looks like from start to finish and to provide E2E test examples with Cucumber and Nightwatch.

We will show real examples of our backyard project, how to set up the environment, and how to automate some of your tests.

The test includes three parts:

  • Feature file, which is a clear description for each step of your test using Gherkin syntax.
  • Step file, which is the script for matching the features and the Java code.
  • TestRun, which is the configuration setup for running your test.

Some important things need to be taken into consideration when you configure your test-runner.ts. You need to configure it for your needs:

  • Add environment option. Tests should be able to be easily rerun in any environment.
  • Plan which tests to be run
    • Smoke tests only?
    • Specified tagged test(s)?
    • Specified feature file(s)?
  • Plan where tests are to be run
    • Chrome in headless mode
    • Browserstag
    • Saucelabs
  • Plan to run tests in parallel (if this is what you need)
  • Set seleniumHost and seleniumPort

Writing nightwatch.conf.ts file

Take screenshots for the failed tests. This will help in debugging the failed tests.

Add option for your tests to be run on Saucelabs and Browserstack. This will help in browser compatibility testing.

Package.json

{

“name”: “api-e2e”,

“version”: “1.0.0”,

“scripts”: {

“report”: “node build/create-html-report.js”,

“prebuild”: “npm run clean”,

“build”: “tsc”,

“build:watch”: “npm run build — -w”,

“clean”: “node ./clean-up.js”,

“lint”: “tslint -c tslint.json   ./src/*ts  ./src/**/*ts”,

“debug”: “node –inspect-brk run-tests.js”,

“test:run”: “cucumber-js –require ./src/features/\\*\\*/*ts –require-module ts-node/register –require ./src/hooks/*.ts –format ./node_modules/cucumber-pretty ./src/features/”,

“test”: “./node_modules/.bin/ts-node src/test-runner.ts”,

“selenium:local-standalone”: “ts-node server.ts”,

“selenium:local-docker-hub”: “docker-compose up”,

“generate-report”: “ts-node ./src/create-report.ts”,

“e2e-test”: “npm test”

},

“keywords”: [ ],

“author”: “”,

“license”: “ISC”,

“devDependencies”: {

“@types/cheerio”: “^0.22.7”,

“@types/cucumber”: “^4.0.4”,

“@types/debug”: “^0.0.30″,

“@types/faker”: “^4.1.2″,

“@types/lodash”: “^4.14.115”,

“@types/node”: “^10.3.2”,

“fs-extra”: “^6.0.1”

},

“dependencies”: {

“bluebird”: “^3.5.2”,

“cheerio”: “^1.0.0-rc.2”,

“chromedriver”: “^2.40.0”,

“config”: “^1.30.0”,

“cucumber”: “^5.0.0”,

“cucumber-html-reporter”: “^4.0.1”,

“cucumber-junit”: “^1.7.1”,

“cucumber-pretty”: “^1.4.4”,

“debug”: “^3.1.0”,

“deepmerge”: “^2.1.1”,

“emailjs-imap-client”: “^3.0.7”,

“faker”: “^4.1.0”,

“fecha”: “^2.3.3”,

“geckodriver”: “^1.12.2”,

“jsonschema”: “^1.2.4”,

“lodash”: “^4.17.21”,

“nightwatch”: “^1.0.11”,

“nightwatch-api”: “^0.4.0”,

“npm-run-all”: “^4.1.5”,

“selenium-server”: “^3.12.0”,

“start-server-and-test”: “^1.7.1”,

“ts-node”: “^7.0.1”,

“typescript”: “^2.9.1”,

“wait-on”: “^3.0.1”,

“yargs”: “^11.0.0”

}

}

Create helpers for your tests

Create html report for the test results

Cucumber uses reporter plugins to produce reports that contain information about what scenarios have passed or failed. It produces pretty HTML reports that help in the visualization of failed and passed tests.

html-reports.ts

var reporter = require(“cucumber-html-reporter”);

 

export function generateReport( ) {

console.log(“generating report”);

var options = {

theme: “bootstrap“,

jsonFile: “reports/cucumber.json”,

output:

“reports/test_” +

new Date( )

.toJSON()

.split(“:”)

.join(“”)

.split(“-“)

.join(“”)

.split(“.”)

.join(“”) +

“.html”,

reportSuiteAsScenarios: true,

launchReport: true,

storeScreenShots: true

};

 

reporter.generate(options);

 

return new Promise(resolve => {

setTimeout(resolve, 0);

});

}

//more info on `metadata` is available in `options` section below.

 

//to generate consodilated report from multi-cucumber JSON files, please use `jsonDir` option instead of `jsonFile`. More info is available in `options` section below.

create-reports.ts

import { generateReport } from ‘./html-report’;

generateReport( );

Example of reports can be found here: https://github.com/IT-Labs/backyard/tree/master/doc

E2E test examples with Cucumber and Nightwatch

In this article, we are providing examples of how we can automate some testing scenarios.

We are going to show how we implemented the following scenarios:

  • User access tests
  • Search list item
  • Edit Item
  • Create item
  • Delete item
  • Test empty system

 

The following diagram is describing our application, where we wanted to simulate a complex application structure that depends on different services and components.

https://github.com/IT-Labs/backyard/blob/master/doc/Backyard%20arch.jpg

As we discussed in our previous articles, we are using dockers and volumes to run our test in repetitions. All commands are written in shell scripts for an easy start. For example, starting all applications for testing all scenarios you should only run the restart.sh file and system are ready for use, cleaning all resources you should use clean.sh.

Steps:

  • Download code from https://github.com/IT-Labs/backyard
  • Build the code with the script build.sh
  • Start the environment with a restart.sh ./path to your local back yard folder
  • Run tests with a test.sh

More details about the commands and environment you can find in the project readme file https://github.com/IT-Labs/backyard/blob/master/README.md.

Here is one example of how to run multiple tests with the previous steps:

https://raw.githubusercontent.com/IT-Labs/backyard/master/doc/TestRun.gif

User access rights

In this example system, we have two types of users: public and administrator users. For testing both scenarios we have two feature files, https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/home/home.feature  and  https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/home/home-public.feature .

Here is how gherkin is written:

Scenario Outline: Verify menu items from Home page when user is admin

    Then the menu <Link> should be <isVisible>

Examples:

| Link        | isVisible |

| Home        | true      |

| Admin       | true      |

| About       | true      |

| Hi Username | true      |

| Login       | false     |

| Logout      | true      |

 

Code:

Then(/^the menu (.*) should be (.*)$/, function (link:string,isVisible:boolean) {

return menu.checkLinkVisibility(link, isVisible);

});

Code related to this example  https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/home/home-steps.ts   , https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/domain/menu/menuPage.ts

Example run: https://github.com/IT-Labs/backyard/blob/master/doc/featureHomeAdminRun.gif

Search list item

For this example, we prepare one scenario, where we are going to validate the search by name. For that purpose, our volume data has already populated data.

Code can be found here: https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/items/items.feature

https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/domain/items/itemsPage.ts

Scenario: Verify correctness by Search by Name

When I add SearchItem in items search name

Then I should get item with that SearchItem as a result

When I add Search in items search name

Then I should get 1 results that contain Search in the name

 

Example from the run https://raw.githubusercontent.com/IT-Labs/backyard/master/doc/SearchItem.gif

Delete item

In this example, we are presenting how we can test deletion of one item from out test environment. For that purpose, our volume data has already populated data. We must be aware that for each test run we must restore the volume from initial backup.

Code can be found here: https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/items/delete-items.feature

https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/domain/items/itemsPage.ts

Scenario: Verify delete item

When I click on Delete icon for item with name DeleteItem

Then I should see delete confirmation modal

When I click on OK on the modal

Then Item with name DeleteItem is not shown on items page

Example from the run: https://github.com/IT-Labs/backyard/blob/master/doc/DeleteItem.gif

Create item

In this example, we are presenting how we can test creating an item. For this test requirements are the item not exist in the system, and for that purpose our database before each run is restored from initial backup.

 

Scenario: Verify user can create item using all fields in Create item page

When I click on create button

Then I should see create item page

When I add Name SampleName

And I add Description SampleDescription

And I select Status

And I check Is Public checkbox

And click create item

Then Item should be created

And I should be redirected on list items page

Code can be found here:  https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/items/create-items.feature

https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/domain/items/itemFormPage.ts

Example from the run: https://github.com/IT-Labs/backyard/blob/master/doc/CreateItem.gif

Edit item

In this example, we are presenting how we can test deletion of one item from our test environment. For that purpose, our volume data has already populated data. We must be aware that for each test run, we must restore the volume from initial backup.

 

Code can be found here: https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/items/edit-items.feature

https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/domain/items/itemsPage.ts

 

Scenario: Verify user can edit item changing only mandatory fields

When I click to edit item with name EditItem

Then I should see edit item page

When I change the Name to TestEditedItemName

And click create item

Then I should be redirected on list items page

And  I add TestEditedItemName in items search name

And I should get item with that TestEditedItemName as a result

Example from the run: https://github.com/IT-Labs/backyard/blob/master/doc/EditItem.gif

Test Empty system

This is a scenario where we are verifying how the system will behave when there are no records. For that purpose, we have a separate environment where a database is without any record, and we start all service as separate instances.

Code can be found here: https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/home/home-public.feature

https://github.com/IT-Labs/backyard/blob/master/fe/e2e_tests/src/features/home/home-steps.ts

 

Scenario: Verify Home page main page when there are no public published items

Then I should see message in the middle of the screen: There are no items yet!

 

Example from the run: https://github.com/IT-Labs/backyard/blob/master/doc/emptySystem.gif

What not to do when writing E2E tests

  • Don’t Write Them Without Defining the Reason
    • Automated acceptance testing (an encapsulation of customer expectations)?
    • Automated regression testing (preventing regression errors)?
    • Automation of smoke test flow?
  • Don’t Duplicate Coverage
    • If the flow can be covered with unit test, don’t write E2E test for it
  • Don’t Use a Single-Layer Architecture
  • Don’t Use Breakable Selectors
  • Don’t Expect Your Suite to Be Maintenance-Free
  • Don’t Ignore Flaky Tests

Summary

It is so important to test your software from top to bottom. E2E testing is a great way to do it — you just need to follow some rules that will implement the process properly.

When you write the tests, add a descriptive test step. A descriptive test step is helpful when a test fails, it gives them a clean and readable description of test steps (for example why this test, how it is done, etc.). Also, descriptive tests simplify the process of automation test maintenance  in large projects. What’s more, having reports that will visualize and tell you which test failed and which test passed is will also help.

Since our approach implements feature files, you need to keep into consideration that the feature files can easily become very large and messy files. And running the tests — especially if they are part of some CI/CD process — can take considerable time, especially when you want to run your test in different browsers and you have large coverage.

On the other hand, large coverage is always good to have no matter the duration of the tests, because it creates greater confidence in your code.

Jovica Krstevski
Technical Lead at IT Labs

Aleksandra Angelovska
Quality Assurance Lead at IT Labs

June 2021