ACA Blog

ACA Blog


August 2021
M T W T F S S
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Categories


Automated testing: make business & IT love each other

Stef NotenStef Noten

Automated testing, when done right, eliminates the increased risk and additional problems manual testing introduces. In this blog post, we disprove the misplaced belief that manual testing and evidence collection adds a lot of value.

At ACA, we embrace an agile methodology. As such, we work closely together with our customers to deliver solutions that solve their actual business problems. Moreover, we strive to solve them fast. This approach has shown its value time and time again, because it captures as much feedback as soon as possible.

Some customers historically enforce thorough manual user acceptance testing right before a software change can go live. The reason is obvious: their business greatly depends on this software, so any risk and financial impact resulting from it should be ruled out. So manual testing after finishing all development makes perfect sense, right?

Or not? Counterintuitively, our team has often witnessed that manual testing indirectly causes more risk and problems, not less! We would even go so far as to say it’s like a virus. A virus that infects every part of the organisation, often completely unnoticed! How is this possible? Because the manual testing isn’t just the final check it was meant to be. It’s a process. A process that steers the way you do change management, the way you plan resources and the way you assign and shift accountability.

Manual testing: higher costs and risks

Scheme that shows how manual testing leads to either a higher development cost or increased risk.

Perfectly clear, right?

Maybe not. Let me summarize:

  1. Because manual testing is time consuming, it results in too few and therefore too big releases. This causes:
    • Risky go-lives
    • Slow feedback
    • Tricky dependencies, complicating hotfixes
  2. Due to the additional safety net, accountability shifts away from the development team. This inevitably leads to missing automated test scenarios and checks, causing:
    • Insufficient confidence for future go-lives
    • Mandatory (or missing) manual regression testing with future changes
    • Bugs (over time)

All of this costs real money and introduces real risks!

Putting automated tests in the driver’s seat

Instead of trying to tackle each issue at the leaves, we tried to take a relatively small action to tackle the problem at the root. We tried to eliminate the misplaced belief that this type of manual testing and manual evidence collection adds a lot of value.

We already had a huge amount of automated tests as part of our test-driven development lifecycle. So how could we start leveraging these? We started by explaining our current testing habits to the business users. Then for a concrete project, we started defining scenarios together and finally we provided them with continuous insight in the results.

Scheme that shows types of testing and how important manual testing is still thought to be

Step 1: teach the business about automated testing

Which kinds of automated tests are already in place? Which guarantees do they provide? And which guarantees do they not provide?

The business deals with their business. They don’t know application architecture, never mind what an automated test actually is. You can’t see (most) automated tests, so how could you ever trust them? As a result, reducing reliance on manual testing is not about changing some practices and adding some checks. At the core, it’s about providing insight and earning trust.

More specifically, we explained the following:

Our business domain layer is completely unaware of any technology or infrastructure, so those things can’t possibly break the guarantees given by the tests at this layer.

Our business domain layer is completely unaware of any technology or infrastructure, so those things can’t possibly break the guarantees given by the tests at this layer.

Our use case layer relies on infrastructure only in high-level business terms, e.g. on something that can ‘save’ or ‘find’ things. The tests guarantee correct results for the actions and queries in this layer, given correct infrastructure.

Our use case layer relies on infrastructure only in high-level business terms, e.g. on something that can ‘save’ or ‘find’ things. The tests guarantee correct results for the actions and queries in this layer, given correct infrastructure.

Concrete technology/infrastructure choices are plugins at the borders of our applications, in the same way that a printer is plugged into a computer: Excel does not know the model of your printer, it just knows that it can print. If the plugins (‘printer’) work, the use cases (actions and queries) will also work.

Concrete technology/infrastructure choices are plugins at the borders of our applications, in the same way that a printer is plugged into a computer: Excel does not know the model of your printer, it just knows that it can print. If the plugins (‘printer’) work, the use cases (actions and queries) will also work.

We also test all layers together using click-through-UI tests. As they can be shown visually and cover all parts in integration, these tests provide a lot of confidence for the business.

There are some gaps in our automated testing strategy. Some integrations with infrastructure or external services can’t be verified easily in an automated way. This is where manual testing needs to be focused.

The business should not waste their time by checking whether a certain message appears when you click on a certain button.

The purpose is not to turn business users into developers, but to show them why the trust of developers in these tests is self-evident. So instead of overwhelming them with technical details, ask what they are concerned about and provide answers patiently. They won’t remember all the details, nor should they. But the trust will stick. And only after earning trust, they’ll accept changing testing habits. Even more so, we found them asking for it themselves!

Step 2: Determine test scenarios together

Which scenarios are business critical? Which checks are necessary to provide sufficient confidence to go live? These are questions we now address during story elaboration. Consequently, the resulting automated tests now have become a deliverable of each story.

Manual tests often focus on long, end-to-end business processes. Even if each part of the process was proven to work perfectly in isolation and you were the person to flip the switch in production: would you feel 100% confident? For those brave enough to answer yes: it’s either because you know everything in detail, or you don’t know enough!

As such, we focused on the same end-to-end business flows, simulated by clicking through the integrated application (excluding only some minor external services).

Step 3: Provide continuous insight

The right automated test scenarios were now decided upon, but how could the business users know that these had been implemented correctly? And how could we provide them with evidence, which they were required to document? Therefore, as a last step we introduced a detailed test report, showing all executed steps in Given-When-Then form.

Image of a report showing all executed steps in an automated testing cycle

Clicking on a step opens a popup with screenshots and a log of all executed checks. This provides our business users with evidence and confidence that the correct things have been checked.

Image of an automated testing report showing a log of all executed checks. This provides business users with evidence and confidence that the correct things have been checked.
Image of an automated testing report showing a log of all executed checks. This provides business users with evidence and confidence that the correct things have been checked.

✅ Positive effects of introducing automated testing

We introduced this new way of working together about 6 months ago, for multiple waves of development on a new application. Throughout this project, we have noticed a lot of positive effects:

⚠️ Pitfalls

Even though we are incredibly positive about this new way of working, there are some remaining pitfalls to look out for.

➡️ Next steps

We feel this new way of working is a huge improvement, but it is just the beginning. We see the following next steps:

What do you think? What do you (dis)like about our approach? We’d love to hear from you in the comments below!


More cool software development blog posts:

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x