Improving Testing of AI Systems with Grey-box Testing Technique
2019-01-29
6:00 p.m.
The Albany Club - 91 King Street East, Toronto, Ontario
Yury Makedonov
Yury Makedonov was trained as a researcher and worked in a Research and Development institution, dealing with composite materials. He has a Ph.D. degree in Physics and Math.
He is not a rocket scientist anymore. Now he is using his skills and knowledge to improve software quality. Yury has over 20 years of testing experience from small startups to large companies and government organizations and recently has been working as a QA Manager, a Test Manager, and a consultant.
Speaker: Yury Makedonov
There are two main challenges to testing systems that incorporate elements of Artificial Intelligence (AI). The same input can trigger different responses while an AI system adapts to a new environment. It may also be challenging to understand what is the correct response.
Such behavior breaks one of the main principles of traditional testing – repeatability of execution of test cases. This is akin to shooting a moving target and not knowing whether you missed. Testers feel frustrated when their traditional approach can’t be used anymore and have no confidence in the outcome of their testing.
Yury Makedonov explains that to successfully test an AI system, you need to have direct access to the system’s state. Using an example of a simplified machine learning system with a few simple models of its internals Yury shows how to use the state of this system for effective testing and how to deal with test data, the most difficult part of AI testing. You can apply these “grey box” testing techniques to a wide range of AI systems from simple machine learning systems to complex neural networks.