GUI testing using computer vision

Tsung Hsiang Chang, Tom Yeh, Robert C. Miller

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

152 Scopus citations

Abstract

Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development.

Original languageEnglish
Title of host publicationCHI 2010 - The 28th Annual CHI Conference on Human Factors in Computing Systems, Conference Proceedings
Pages1535-1544
Number of pages10
DOIs
StatePublished - 2010
Event28th Annual CHI Conference on Human Factors in Computing Systems, CHI 2010 - Atlanta, GA, United States
Duration: 10 Apr 201015 Apr 2010

Publication series

NameConference on Human Factors in Computing Systems - Proceedings
Volume3

Conference

Conference28th Annual CHI Conference on Human Factors in Computing Systems, CHI 2010
Country/TerritoryUnited States
CityAtlanta, GA
Period10/04/1015/04/10

Keywords

  • gui automation
  • gui testing
  • test by demonstration

Fingerprint

Dive into the research topics of 'GUI testing using computer vision'. Together they form a unique fingerprint.

Cite this