Why do we need automation test sites? 

As many automation frameworks/tools have risen in the last few years, we have more options to choose from when we evaluate which tool is best suited to a given project, and of course, practice makes perfect. 

Practice requires environments prepared for that purpose, and that implies choosing a demo web app that allows us to add data and test workflows without affecting real users or any other collateral effect, playing the role of “Application Under Test”. This demo web app also supports us when preparing a demo or learning a new framework/tool to automate tests or, while assessing a new automation framework to help us decide which is the best one for a given project. 

In this opportunity, we are looking for that project to use as a model (scenarios to test) for our whole QA team, that allows us to build automation suites using different frameworks, starting with Playwright and Cypress. The outcome of this exercise will be used as our IT Labs automation playground. Ideally, this project is open-source and allows us to build an immutable environment, working locally or deploying it in our in-company environments and, if needed, to add improvements to source code in order to achieve a good selectors strategy and add features to increase our testing coverage.  

What are we looking for in automation test sites? 


Usually, we look for a few elements (commonly found in all web apps) as:  

Login/Authentication: to test features which require to be logged as a valid user (right permissions applied) vs public features (available with no authentication needed).
Roles/Authorization: testing features under different profiles and authorization levels (right permissions management). For example: actions that can be performed as admin users but not allowed to regular users.
Forms: Save information and verify that information has been saved accurately.
Modals/Popups: to check success/error messages, alerts, or modal forms to save information.
Upload content: upload content and verify they have uploaded correctly and are available to consult/download such as files and images.
Download content: testing that we can download content generated by web app, such as invoices, reports and so forth.
Mouseover/Tooltips: to check help messages or error messages.
Tables/Grids: to test/search information as products listings, employees, courses and so forth, organized in table or grid formats.
Localization: to test dates and currencies formatting, time zones and languages which depend on countries/region. Also, features that should be available in certain countries/regions.
API: we can test features through REST request, as they are faster than UI and, also, we can automate these requests.
Selector Strategy: when we automate user interface tests, the first thing is to “find” elements in our web app, and then, we chain actions to those elements; this is how we interact with the web app. A selector strategy is the way we “find” and interact with all elements. So, this web app also will allow us to establish a good selectors strategy. Most elements have proper ids and names (which represent the data as username, password, personalID, address, save and submit, instead of IDs randomly generated), class names, attribute data-test or data-test-id (usually do not change over time).
Workflows (e2e use cases): a workflow is a pattern of activity enabled by the systematic organization of resources into processes that transform materials, provide services, or process information. We look for workflows because we do not test isolated components; we test workflows just like our users would. Some examples are banking features (deposits, withdrawals, transfers, create accounts), e-commerce (add products to carts, accounts creation, set addresses and payment methods), human resources (creating employees, requesting PTO, generating reports), and so forth.
Easy to manage (install, run, stop and restart) as our purpose is to work primarily in an automation suite and not get distracted in other kind of issues.
Easy to reset (reverting back to initial state) in order to get predictable test outcome consistently (always).
Easy to maintain in case we want to add more challenging features to be tested.

Automation test sites assessment 

We compared elements and features to determine the complexity of workflows and the selector strategy we can use in a few demo sites created for testing purposes:   

Ecommerce: SauceDemo, Juice Shop and Automation Exercise    

Banking: XYZ Bank, Cypress Real World App and Parabank  

Human Resources: OrangeHRM  

Flights and hotels booking: PHP Travel 

  SauceDemo  XYZ bank  Automation exercise  Parabank  Orange HRM  Juici Shop  PHP Travels  Cypress Real World App 
Login/Authentication  Yes    Yes  Yes  Yes  Yes    Yes 
Roles/Authorization    Yes  Yes    Yes  Yes  Yes   
Forms (save & check)    Yes  Yes  Yes  Yes  Yes  Yes  Yes 
Modals/Popups    Alerts  Yes    Yes  Yes    Yes 
Upload content (files, images)          Yes  Yes     
Download content (files)      Yes        Yes   
Mouseover/Tooltips        Tooltip    Yes  Yes   
Tables/Grids  Grids  Yes    Yes  Yes  Yes  Grids  Grids 
Localization (other than English)            Yes     
Accessibility                 
API      yes      Yes     
Good selector strategy            Yes    Yes 
Good workflow  Yes  Yes  Yes  Yes  Yes  Yes  Yes   

With this assessment phase, we arrived to the final two candidates  Orange HRM and Juice Shop. After further assessment, we chose Juice Shop because it has almost all the features except download files (files are open in another tab or window).  

IT Labs Automation Playground 


An automation playground is a plus when we want to assess different frameworks (writing and running the same test in different frameworks), because it allows us to focus our efforts on the technical capabilities and pros/cons of the tool/framework under assessment with complete impartiality as our Application Under Test (AUT) will be the same. 

We assess in an automation framework features different than the ones we look for in an AUT, such as: 

  • Support for multiple browsers. 
  • Support for interaction with elements in the web app, reading attributes as so on described before as testing strategy. 
  • Support for asynchronous. 
  • Multiwindow support. 
  • Download/upload of content. 
  • Visual testing. 
  • Reporting. 
  • Integration with different CI/CD tools. 
  • Error handling. 
  • Retrying (when fails). 
  • Mocks. 
  • Assertions. 
  • Support for Page Object Model or other patterns. 
  • Organizing suites. 
  • Parallelism. 
  • Parametrization. 
  • Debugging. 

As Juice Shop is an open-source project, you can fork it, as we did with our own automation-playground repository and add new features (such as download files, that it is missing in the original project right now). In that way, our automation playground will not be affected if the project is not available online anymore, and we will not affect any real user or any other side effect.  

We will use this to compare different automation frameworks as we would use the same site to do the test (with most of the features we want in automation test sites). 

We can test:   

  • Login with email and password; and using a Google account as well.  
  • A role for regular users and a role for deluxe membership where you will get discounts and delivery for free.  
  • Upload pictures as photo wall or setting user profile.  
  • Localization as there is a lot of languages available.  
  • Different workflows in a shopping cart such adding products, setting payment details, adding addresses, add reviews to products, send a complaint, a chat for customer support, order history and recycling box.  

Modals and popups, forms, tooltips, tables, grids with IDs, aria-label and placeholders are available to test interactions. We can implement a selector strategy based on IDs, names, css classes, text, links, and attributes. 

Juice Shop is a project created for security testing purposes, so, we can implement security and API testing as well. A good balance to practice automation testing. You can learn more about the challenges here. 

Our discover in our playground 

AS we mentioned before, there are many frameworks to automate tests and we need to choose the one which fits best for our project/product. 

After running a few basic tests in Playwright and Cypress, we can verify the support in following items for each framework: 

Features  Cypress  Playwright 
Support for multiple browsers  Yes  Yes 
Support for interaction with elements in the web app, reading attributes and so on (described before as testing strategy)  Yes  Yes 
Support for asynchronous  Automatic  Yes 
Multi-window/multi-tab support    Yes 
Download/upload of content  Yes  Yes 
Visual testing  Yes  Yes 
Reporting  Yes  Yes 
Integration with different CI/CD tools  Yes  Yes 
Error handling  Yes  Yes 
Retrying (when fails)  Yes  Yes 
Mocks  Yes  Yes 
Assertions  Yes  Yes 
Support for Page Object Model or another pattern  Yes  Yes 
Organizing suites  Yes  Yes 
Parallelism  Yes  Yes 
Parametrization  Yes  Yes 
Debugging  Yes  Yes 
API testing  Yes  Yes 

Playwright 

Playwright can be installed with just 1 command:

Playwright Code Generator

We can start recording a test with 1 command as well (we will bring more about it in future articles):

We can record tests, export them in several languages to include them in our suites.

Selectors Strategy

This is an example of multiple selectors that Playwright allows.

We can also build our selectors which depend on data or generated classes:

Page Object Model in Playwright 

Let’s see an example of login: 

Following the page object model (POM) we have a page login-page.js and the spec file ShoppingCart.test.js where we are using the login page: 

Login-page.js 

ShoppingCart.test.js

Cypress

Cypress can also be installed in a few commands:

It has a user interface where you can configure it, run the tests and inspect elements. You can activate it with the command: 

This is how the user interface looks like:

And this is how a running test looks like: 

Selectors Strategy

This is an example of multiple selectors that Cypress allows. We can also build our selectors which depend on data or generated classes:

Cypress Studio (experimental)  

Page Object Model in Cypress

Let’s see the same example we saw above: login. We have a page LoginPage.js and the spec file shopping_cart.cy.js where we are using the login page. 

LoginPage.js

shopping_cart.cy.js

Speed of Tests

It’s important to mention that both frameworks work without Selenium and Webdriver, they tend to be faster. Cypress runs the tests in 25secs, and Playwright runs the same in 15secs, both in headless mode.
You can check the suites here: Playwright and Cypress.

Conclusion


There are many websites out there (both online and available to be installed locally -open-source projects-) and choosing a tool to learn and practice, or evaluate an automation tool, will be based on your purpose and need. 

It is also worthwhile when we want to compare several testing frameworks and create test suites using different tools and approaches but the same website. In that way we can focus our efforts on the technical capabilities and advantages of the tool/framework under assessment with complete impartiality. 

You need to make sure that the playground project you choose allows you to establish a good locator strategy and a good workflow to be tested (not only interacting with web elements) because from a real-world user’s perspective we do not test components isolated. 

We can use Juice Shop as our playground web app to evaluate different test automation frameworks and to establish our test strategy with the advantage of having an immutable environment, easy to manage and reset guaranteeing consistent outcomes over time (same results in all testing execution cycles). 

The final takeaway is that you need to choose the automation framework that fits best for your project, and this will be determined by the kind of project and features and complexity you need to test. 

Both Playwright and Cypress are easy to install, have wide support from communities, and have all that we usually look for in automation frameworks. They are fast because they are not based on Selenium and Webdriver but still powerful in debugging and integration with several CI/CD tools. Definitely good choices for bringing automation regression into our projects/products. 

Author:

Maria Aguilera


Senior Quality Assurance Engineer in IT Labs