One year with E2E tests – A recap

We, team Smart Alarm, started implementing the first end-to-end tests about a year ago to ensure that our application's critical functionalities and workflows are behaving as expected. At the time of writing this article, we had implemented almost 300 of them – so it’s time to recap our experiences so far!

We will show you what E2E tests are and why we needed them to begin with. We will give an overview of common techniques and share the experiences we had during the implementation. In an upcoming article, we will also show you how we integrated our tests into our build pipeline.

We always aim to deliver new features and bug fixes to our customers as soon as possible, but ensuring the quality of our releases has proven to be time-consuming in the past. We had a QA session for each release – meaning multiple developers sitting in a room, processing and ticking off a predefined list of manual tasks relating to the software, to make sure everything was working as expected. This list includes items such as:

Enter wrong credentials. Expected: An error message is shown.
Enter correct credentials. Expected: You are logged in and directed to the start page.
Click ‘Logout’ button. Expected: You are redirected to the login page.

This list was already pretty long, and it grew with every new feature we develop. When a critical bug was found during a QA session, the release was delayed until the bug was fixed.

Developers are people, and people make mistakes – a refactoring you are working on might affect the application in ways you did not think of. The risk of side effects increases with complexity. It's even more annoying when it takes up to two weeks (until the next QA session) for someone to identify the issue. The release is delayed and customers have to wait longer for new features.

Wouldn’t it be great to be notified of these major bugs before the breaking changes are even part of the release candidate? Wouldn’t it be great to automate at least parts of those manual tests?

Absolutely! What we need is end-to-end testing, or E2E tests for short. The idea is to set up the whole application – including all the necessary subsystems like the database and API backend – as close as possible to the production environment. We want to run common user workflows by simulating clicks in the application – in an automated way, as part of our build pipeline.

Unit and module tests are, of course, a first step towards ensuring the product's quality. Unit tests are the base of the so-called test pyramid. Yet the more complex your application gets, the more difficult it gets to predict the side effects your change could cause when you put all the modules and subsystems together.

E2E tests are not a new concept; they have been around for a long time. As a result, many solutions are available. When we chose an E2E testing tool, our requirements included:

Good documentation and an active community
Easy setup
Useful test reports
Stable tests – no brittle tests and no false positives
Tests are easy to write, read, and maintain

The most common tool is probably Selenium, which was first released in 2004. It supports the most popular web browsers and operating systems and can be used with a wide range of programming languages. Protractor, Angular’s long-standing default E2E testing tool, is also based on Selenium. However, it has been announced that Protractor will no longer be developed from the end of 2022.

Another popular testing tool, Cypress, was released in 2017 and supports Chrome, Firefox, and Edge, covering the majority of available web browsers. Tests need to be written in JavaScript (or TypeScript), but that is fine for us.

We experimented with both Cypress and Protractor (which still was an active project back then), implemented some basic tests and tried to integrate them into our build pipeline. The Cypress tests were immediately stable, and its installation could not be easier. The Protractor tests were brittle and their setup pretty complicated in comparison to Cypress. Also, the Cypress tests were more readable.

In the end, we picked Cypress. Besides the reasons mentioned above, we were impressed by some of its more modern features. In particular, the “no manual wait” guarantee, time travel and debugging features are extremely helpful.

Install cypress

Installing Cypress for your project is as easy as

1 npm install cypress --save-dev
2
3 # OR if you are using yarn
4
5 yarn add cypress –dev

In order to open the cypress test runner, you type

1 npx cypress open
2
3 # OR if you are using yarn
4
5 yarn run cypress open

The test runner lists all your existing tests and detects the browsers on your OS. You can select a browser and run single tests just by clicking on them.

When first opening the test runner, Cypress will create a folder called cypress in your project root with some subfolders in it. You can ignore them for now, as we’ll come back to some of these subfolders later.

Using TypeScript

When it comes to readability, using TypeScript instead of plain JavaScript is a good option, especially if you are using it anyway in your project and you normally wouldn’t want to use another language in your tests. You can enable TypeScript in your Cypress tests too: just add a tsconfig.json file in your cypress folder.

1{
2  "compilerOptions": {
3    "target": "es5",
4    "lib": ["es5", "dom"],
5    "types": ["cypress"]
6  },
7  "include": [
8    "**/*.ts"
9  ]
10}

All test files are located in the ‘Integration’ subfolder within your Cypress folder. In the following test snippet, we open our application, enter some invalid credentials, and verify that an error message is shown.

Note that after clicking the button, your application might take some time until the error message appears, as we need to check the credentials with the backend first. If you already know Selenium, you might expect some sort of wait logic here. Cypress, however, does all that waiting for us automatically!

There is a lot more that Cypress can do, but this would probably fill a bunch of blog articles on its own. Head over to https://docs.cypress.io/ for more information.

When you run a test using the test runner, you can sit back and watch your application run the defined workflow. All actions (like clicks or requests) are shown in the left panel. Once the tests are finished, you can even select the individual steps and see the state of your application at any given time. Handy!

Best practices

While the test shown previously will work perfectly fine, it is not a good test as it violates some of the best practices commonly used for writing tests. Best practices are great in general, but following them blindly might not always be the best for you and your application. We had to gain some experience ourselves to learn what works for us.

Define a baseUrl!

Remember the line

1cy.visit('http://localhost:4200');

in our previous test? What if you want to test a remote installation? What if your colleague runs the application on a different port? He would have to change the URL in every single test file. You can solve this problem by configuring the base URL in your Cypress config, cypress.json in the project directory by default.

1{
2  baseUrl: "http://localhost:4200"
3}

Now, when you want to visit your application, you just need to call

1cy.visit('');

The configured base URL will be prepended.

You can configure all sorts of variables in the config file. For example, you might also need to access to your backend URL in your tests. You can simply put them in your environment like this:

1{
2  "baseUrl": "http://frontend",
3  "env": {
4        "api_url": "http://localhost:9393",
5  }
6}

The configured values are available in your tests, e.g. using Cypress.env('api_url')

Use data-cy attributes

Maybe you already know the concept of data attributes in HTML: Using data attributes, you can store additional information on HTML elements. But how does that help us when writing tests?

There are different ways to select the elements on your page, like by (HTML) id, CSS class, or text content. The standard recommendation is NOT to couple your selectors to brittle definitions like styling details, but rather add a dedicated data-cy attribute to the elements instead.

In our team, we follow this principle sometimes, but not all the time. In some cases, it just does not seem worthwhile to bloat the templates with additional attributes. For example, the appearance of our login page has not changed for ages. Even if it does change in the future, we only have one place where we need to update anything - which actually brings us smoothly to the next topic, the…

Page object pattern

You will probably not have only one single test for the login page, as above. You will test multiple scenarios, entering valid credentials, wrong credentials, admin credentials, and so on. Repeating all the element selectors like cy.get('#username') might bloat the test code and reduce readability. If your login process changes, you will spend a lot of time just updating the selectors in your tests.

Therefore, it is highly recommended to extract all page-specific logic and interactions with an element to so-called page objects. For our project, we created an ‘Objects’ folder within the Cypress folder and put all of these page object classes there. For example, the page object for the login page might look like this:

1export class LoginPage {
2
3  login(username: string, password: string) {
4    this.userNameInput.clear().type(username);
5    this.passwordInput.clear().type(password);
6    this.loginButton.click();
7  }
8
9  get userNameInput(): Cypress.Chainable {
10    return cy.get('#username');
11  }
12
13  get passwordInput(): Cypress.Chainable {
14   return cy.get('#password');
15  }
16  
17  get loginButton(): Cypress.Chainable {
18    return cy.get('#login-button');
19  }
20  
21  get errorMessage(): Cypress.Chainable {
22    return cy.get('.validationError');
23  }
24}

Stubs vs real calls and test independence

No matter what your application does, there is probably also some kind of data in the background. This can be rather challenging to handle in E2E tests: To test some specific feature, especially an edge case, you need the data available when running the test.

Cypress allows you to stub your API fully. We used this feature once in our tests, as creating one real specific state in our application may be very time-consuming and thus disrupt the subsequent tests.

In order to stub an API, locate the ‘Fixtures’ subfolder in your Cypress folder. Create a file there, e.g., periodic-check-true.json, and enter the response content you need for your test case, e.g.

1{
2  "retry": true
3}

In your test, enforce the following response, for example:

1 cy.intercept(
2  'GET', 
3  `${Cypress.env('api_url')}/periodic-check`, 
4  { 'fixture': 'periodic-check-true.json' }
5 )

Whenever your application calls your APIs/periodic-check endpoint while running the tests, the defined fixture is then returned instead of the ‘real’ data.

However, for most tests we did not want to cut out the backend, which is an essential part of the software, so we decided to use an appropriately populated test database.

This has some drawbacks, of course. Most of all, a test that runs against a real database changes the underlying data. This can make it difficult to run individual test cases independently from each other: the second test case might behave differently depending on whether the first test case is executed or not. Therefore, you either have to ensure before each test that the underlying data are in the expected state, or you are fine with the fact that a test may potentially be influenced by others, and you cannot execute it without running the previous one first.

In a way, we have a mixture of both – we probably still have some learning to do here. Ensuring the preconditions for a test can be time-consuming (in development as well as in runtime), so we decided in favor of a trade-off – at the beginning of a test file, we make sure that the preconditions are met; between single tests within one file, we usually don’t.

… and others!

There are many more recommendations for writing tests. Again, the Cypress docs are a great source of knowledge.

The most prominent drawback for us is definitely the long build time. As we will explain in an upcoming article, we integrated our E2E tests in our CI pipeline and all tests need to pass to continue with the release process. At the time of writing this article, the E2E tests alone have a runtime of around 30 minutes.

Clearly, implementing new features now takes longer. Writing the E2E tests adds to the development time itself, no matter how good the testing tool is. Our product is already quite mature, so investing this additional time makes sense for us. However, for a new product that is still changing considerably or for an MVP that requires quick results to show to the customer, running E2E tests is probably not the best choice.

Although Cypress is a very good tool, we still run some brittle tests from time to time. Of course, we complain a lot when this happens, as people do. On the other hand, it feels good to have a kind of safety net in our daily work.

Our E2E tests even revealed instabilities that we would probably not have found otherwise: As the click simulation is a lot faster than a user normally clicks, we were able to reproduce side effects that happen in day-to-day work only from time to time, which is annoying enough for the customer, or under difficult circumstances, e.g. a very slow network.

Consequently, we still think E2E testing is an excellent tool for increasing your application’s overall stability. However, it is important to keep in mind that E2E tests do not render manual tests obsolete. Some things that are obvious to the human eye are difficult to test for with software – whatever E2E test tool you use, it won’t tell you that a header looks funny because it is pink text on a red background or that some elements are squashed together because padding is missing. In other words, you'd probably still want to check the overall appearance of your application with human intuition.

We also continue to have our QA sessions. We still find bugs in those sessions. Now, though, they are mostly small UI bugs that don’t have to be fixed immediately. Critical bugs that delay our release deployment happen much less often, which is a massive win for us.

Simone Schäfer

Frontend Developer

SMS digital GmbH

One year with E2E tests – A recap