A tester's thoughts on Automation and AI: 2

In part 2 of the series, Eva Holmquist writes about the exciting possibility of AI called Cognitive QA and how it can be used for a lot of things.

One exciting possibility is to use AI to help with testing activities. This is called Cognitive QA and it can be used for a lot of different things. For instance, we can use Cognitive QA to tell us which test cases, we should run after a change. In that case, it’s important that the system is transparent, so we can determine what criteria the AI uses to determine the best test cases. The answer will be different depending on the criteria. Is the system looking at what functionality was changed? Is it looking at what part of the system is affected by the change? Is it looking at the risk of the change, and how difficult it is to make the change? Is it looking at the probability of bugs and if so, what criteria does it use to determine that? These are just some examples of the different types of criteria that the system can use to determine which tests cases it recommends running. The important thing is that we know the basis of the recommendation. Why were these test cases selected and not others?

One example that I heard during EuroSTAR was a system that used a picture to determine if it was of a wolf or a husky. They trained the system with a lot of pictures. After that, they tried with ten different pictures and the system made the correct choice in nine of them. The problem was discovered when they analyzed the result more in detail and discovered that the system didn’t use the image of the animal at all. The choice, if it was a wolf or a husky, was based on the background!

If we’re using Cognitive QA, we need to test that they determine the result based on the correct criteria. And when we’re using it, we need to be able to determine what criteria it uses. What do you think?

About Eva Holmquist
Eva Holmquist is a senior test specialist at Sogeti. She has worked with activities from test planning to execution of tests. She has also worked with Test Process Improvement and Test Education. She is the author of the book "Praktisk mjukvarutestning" which is a book on Software Testing in Practice.


  • Eva Holmquist
    Eva Holmquist
    Senior Test Specialist
    072-502 83 93