Harnessing AI’s Potential to Detect Visual Bugs Faster

This Industry Viewpoint was authored by Prashant Maloo, Senior Director at Prodapt

Service providers in the Connectedness industry focus on gaining a competitive edge over the competition by maintaining a premium quality of service. However, quality assurance has been a tough challenge, given the need to master the various stages of the product life cycle, such as requirements gathering, product development, product testing, support & maintenance, etc.

In today’s day and age, performing manual visual testing is increasingly challenging. The growing number of visual bugs has been impacting the performance of products, leading to complex problems, including delayed time-to-market, inefficient execution, increased operational costs, depreciated end-user experience, etc.

According to a Gartner report, “61% of enterprise IT leaders responded that end-user experience is critical for application performance monitoring.” Referring to a Browser stack report, “67% of businesses perform visual testing manually to detect visual bugs.“

Here are some major challenges faced by businesses in manual visual testing:

• Lack of a thorough verification mechanism for multiple aspects of the UI
• Prolonged visual testing due to frequent UI changes
• Inconsistency in UI across multiple platforms

To curb this, we recommend an AI/ML-based automation approach to net visual bugs that will accelerate the process of visual testing and improve efficiency by a great prospect.

Fig: AI-driven test automation for visual testing

Implement a three-step approach to AI-driven test automation for faster visual testing

1. NLP-based visual test case prioritization and recommendation
   of the right visual test expert

In this process, the NLP model leverages historical test cases derived from past sprints to assign test case priorities and recommend a visual test automation expert. Historical information containing details such as test case name, test         expert name, test execution time, test parameters, etc., are fed to the NLP model, which studies the information and compares it with the current test cases. Based on the comparison, the NLP model assigns priorities for each new test case and recommends a test expert/s to oversee the testing process.

Recommendations

• Implement NLP engines such as BERT, GenSim, and NLTK to find similar test cases based on visual test case description
• Categorize test cases as ‘High’, ‘Medium’, or ‘Low’ and tag them based on the priority and re-execution of similar historical test cases
• Leverage the human expert associated with similar visual test cases in the past, for scalable and accurate visual test results

 

2. Computer vision (CV) powered visual test case execution for identifying visual bugs

Once the process of prioritizing test cases and recommendation of test experts is complete, implement Computer Vision (CV) powered test execution engine to process images and videos, monitor parameters such as blocked/overlayed UI components, ads blocking UI components, and the responsiveness of visuals across devices, etc. The test expert can choose the kind of testing to be done here from the list of options:

• ‘Snapshot testing’ or ‘Analyze UI Content’ for detecting visual bugs
• ‘Analyze Dynamic Responsive Content’ option for analysis on content placement, content awareness, etc.
• ‘UI color summarizer’ to compare the colors between the UI design and the actual UI

Fig: Computer vision (CV) powered visual test case execution for identifying visual bugs

 

Recommendations

• Perform a stringent pixel-to-pixel comparison for snapshot testing between the UI in production and the UI design
• Leverage techniques like Scale Invariant Feature Transform (SIFT), for inspecting dynamic content such as component alignment across devices, cross-browser testing, etc.
• Implement the UI color summarizerthat uses toolboxes such as Colorthief to reveal the dominating colors, and differences in the UI colors against the design
• Perform UI content analysis, using packages such as Easyocr and Pytesseract to help improve the awareness of UI content, check on the relative placement and differences in the content, etc.

 

3. Automated testing and rapid feedback integration to enable CI/CD

Post AI-driven, automated visual testing, the application goes into deployment, where an analyst keeps a check and provides regular feedback to the development team on the UI changes.

Fig. The complete process of automated visual testing leveraging the power of AI/ML

Recommendations

• Validate automated visual test results with a human test expert recommended by the system
• Leverage Jira for reporting visual bugs and highlighting them for redressal
• Implement Azure DevOps (ADO) for performing automated build and release management 

Conclusion

The three-step strategy helps service providers accelerate visual testing by leveraging AI/ML technology to offer the following benefits:

• 90% improvement in accuracy against manual visual testing
• 60% improvement in test coverage
• Enhancement in the end-user experience and a 75% reduction in operational costs for test case execution

 

I appreciate the efforts of my colleague Sundara Ramakrishnan BL- Manager, Strategic Insights, for his contribution and continuous support in shaping this article.

About the Author

Prashant Maloo

Senior Director, Prodapt

Prashant is a Senior Director – Next Gen at Prodapt, a two-decade-old consulting & managed services provider with a singular focus on the Connectedness industry. He has extensive experience in the telecommunications industry, building products, solutions, and niche technology teams.  He holds demonstrated experience in introducing new technology to solve complex problems and has helped global telecom enterprises achieve new tech adoption through large scale digital transformations. Currently, he is focussed on AI adoption and addressing Cybersecurity needs of the Cloud Network Service domain.

Prodapt is the largest and fastest-growing specialized player in the Connectedness space, serving global firms that ultimately seek to connect people.

This deep focus has enabled Prodapt to provide substantial business value to the largest global telecom, media, and internet firms for over twenty years by transforming their business and technology. Our clients include Google, Facebook, Amazon, Microsoft, AT&T, Verizon, Lumen, Vodafone, Liberty Global, Liberty Latin America, British Telecom, KPN, Windstream, Virgin Media, Rogers, and Deutsche Telekom, among many others.

Our clients see us as valued transformation and operations partners because we understand their technology, business, and market better than anyone. Prodapt is A Great Place to Work certified company with a workforce of over 5000 domain experts across 44 countries spanning the Americas, Europe, Africa, Asia and Australia. Prodapt is part of the 128-year-old business conglomerate, The Jhaver Group, which employs over 30,000 people across 80+ locations globally.

If you haven't already, please take our Reader Survey! Just 3 questions to help us better understand who is reading Telecom Ramblings so we can serve you better!

Categories: Artificial Intelligence · Industry Viewpoint