Options for automated web UI testing.

Web test automation is a large knowledge area, and I would not call myself an expert. But I recently spent some time investigating options for automated UI testing (it was a couple of years since last time), and I thought I’d share my thoughts. I did this investigation because in our current test setup I miss the possibility to run the tests on different devices (mainly Safari on iPhone).

I often have the role of being responsible for a solution from a tech perspective. Making sure that the solution fulfills the business needs, but of course also that it actually works, and continues to work. 

Checking that stuff continues to work in a constantly evolving solution is called regression testing. I usually use automated UI testing to do regression testing on a web app. The tests are usually triggered by our CI/CD solution whenever we deploy a new release to an environment. Being able to quickly check that all critical functionality works, makes me confident when deploying new code, and makes me sleep better at night.

I started doing automated UI testing using Selenium IDE many years ago. I just ran the tests in the browser. It was a fairly low-threshold way of starting with automated UI testing. Since then I’ve evaluated and tested several of the different services/tools available. In periods I have been involved in creating and maintaining tests almost on a daily basis.

There are multiple services/tools that provide similar functionality when it comes to the end result. Meaning, being able to test functionality and/or visual appearance in different browsers. However, the different services have different approaches on how to get to that end result. I (somewhat ignorant) divide these approaches in two main groups:

• Record your interactions in a web browser and add assertions (checking that things are as expected) along the way. These interactions are then replayed when the test runs. There is a UI to simplify tweaking and maintaining your test, as well as adding visual comparison et.c. Simplicity is prioritized over having edge-case features.

• Define your test and assertions in code only. However, some services can import Selenium tests, so you could potentially use Selenium IDE to record and maintain your tests. You might need to install a test framework on your web app, or use specific tools to maintain the tests. Having many features is prioritized over being simple to use.

Being simple to use is my number one priority. If it’s not simple to use, it will most likely not be used at all. I want any developer in the team to be able create or update tests with just a short introduction. I have been working in larger projects where there actually have been a dedicated team of testers, where the budget would allow for using very advanced testing tools.

But what happens when the solution you’re building matures and goes into maintenance mode? The team of testers are gone, and the developers don’t have the knowledge or time to maintain tests, or create tests for new functionality. And this is the time where you need those regression tests the most!

I’ve been looking at several services/tools there are soo many…) and found some that fulfill the “be really simple to use”-requirement:

Ghostinspector (ghostinspector.com) I’ve been using Ghostinspector for years. It’s quite feature rich and is very reasonably priced. The free version will take you quite far, you can use all available features. It’s possible to import and export Ghostinspector tests in various formats. I like that I can export my tests if I decide to stop using Ghostinspector.

Frontend Robot (frontendrobot.com) Frontend Robot is the new kid on the block that aims to be very simple to use. It doesn’t have as many features as Ghostinspector. I also think that the way you work with Frontend Robot when creating/updating tests is much slower compared to Ghostinspector.. Prices for Frontend Robot start as low as 5 USD/month.

Mabl (mabl.com) Mabl also seems simple to use, and has some cool features like “auto-healing” (try to automatically fix tests that broke when the UI was updated). I didn’t actually test it because one feature I need (email testing) is only available in the “Enterprise” level. I don’t want to spend time on “Contact us for a personalized price” (sigh), so I just assume that it can’t match the price for Ghostinspector.

Reflect (refelect.run) Very simple and fast to create and update tests. The tests are also executed very quickly. I really like this tool, and the price is reasonable. But, similar to Mabl, you need to have an Enterprise subscription to be able to do email testing. 

Unfortunately, none of these services have the possibility to do cross-device testing. If you really need automated cross-device testing, and you have the resources for a more complex testing setup, the services provided by Browserstack, Saucelabs, Katalon, or TestProject might be a good fit. But for me, these tools are just not simple enough. 

So, after all this investigation I will continue to use Ghostinspector for automated testing. I’m combining that with occasional manual cross-device testing using Browserstack´s “Live” testing feature.

And just to be clear: I have no affiliation with any of the tools or services mentioned.