TY - GEN
T1 - SikulliBot
T2 - 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014
AU - Kim, Jeeeun
AU - Kasper, Michael
AU - Yeh, Tom
AU - Correll, Nikolaus
PY - 2014/10/5
Y1 - 2014/10/5
N2 - We present SikuliBot, an image-based approach to automating user interface. SikuliBot extends the visual programming concept of Sikuli Script[2] from the graphical UTs to the real world of physical Uls, such as mobile devices' touch-screens and hardware buttons. The key to ou approach is using a physical robot to see an interface, identiI,r a target, and perform an action on the target using the robot's actuators. We demonstrate working examples on MakerBot 3D printer that could move a stylus to perform multi-touch gestures on touchscreen to automate tasks such as swipe-to unlock, playing a virtual piano, and playing the Angry Bird game. A wide range of automation possibilities are made viable using a simple scripting language based on images of UI components. The benefits of our approach are: generalizability, instrumentation-free, and high-level programming abstraction.
AB - We present SikuliBot, an image-based approach to automating user interface. SikuliBot extends the visual programming concept of Sikuli Script[2] from the graphical UTs to the real world of physical Uls, such as mobile devices' touch-screens and hardware buttons. The key to ou approach is using a physical robot to see an interface, identiI,r a target, and perform an action on the target using the robot's actuators. We demonstrate working examples on MakerBot 3D printer that could move a stylus to perform multi-touch gestures on touchscreen to automate tasks such as swipe-to unlock, playing a virtual piano, and playing the Angry Bird game. A wide range of automation possibilities are made viable using a simple scripting language based on images of UI components. The benefits of our approach are: generalizability, instrumentation-free, and high-level programming abstraction.
KW - Automation
KW - Robotics
KW - Tangible User Interfaces
KW - Visual Programming
UR - http://www.scopus.com/inward/record.url?scp=84910058282&partnerID=8YFLogxK
U2 - 10.1145/2658779.2659110
DO - 10.1145/2658779.2659110
M3 - Conference contribution
AN - SCOPUS:84910058282
T3 - UIST 2014 - Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology
SP - 53
EP - 54
BT - UIST 2014 - Adjunct Publication of the 27th Annual ACM Symposium on User Interface Software and Technology
PB - Association for Computing Machinery
Y2 - 5 October 2014 through 8 October 2014
ER -