Say and find it: A multimodal wearable interface for people with visual impairment

Taeyong Kim; Sanghong Kim; Joonhee Choi; Youngsun Lee; Bowon Lee

doi:10.1145/3332167.3357104

Say and find it: A multimodal wearable interface for people with visual impairment

Taeyong Kim, Sanghong Kim, Joonhee Choi, Youngsun Lee, Bowon Lee

Department of Special Education

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Recent advances in computer vision and natural language processing using deep neural networks (DNNs) have enabled rich and intuitive multimodal interfaces. However, research on intelligent assistance systems for persons with visual impairment has not been well explored. In this work, we present an interactive object recognition and guidance interface based on multimodal interaction for blind and partially sighted people using an embedded mobile device. We demonstrate that the proposed solution using DNNs can effectively assist visually impaired people. We believe that this work will provide new and helpful insights for designing intelligent assistance systems in the future.

Original language	English
Title of host publication	UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology
Publisher	Association for Computing Machinery, Inc
Pages	27-29
Number of pages	3
ISBN (Electronic)	9781450368179
DOIs	https://doi.org/10.1145/3332167.3357104
State	Published - 14 Oct 2019
Event	32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019 - New Orleans, United States Duration: 20 Oct 2019 → 23 Oct 2019

Publication series

Name	UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology

Conference

Conference	32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019
Country/Territory	United States
City	New Orleans
Period	20/10/19 → 23/10/19

Bibliographical note

Publisher Copyright:
© 2019 Copyright is held by the owner/author(s).

Keywords

Assistive system
Mobile interface
Multimodal wearable interface
Visual impairment

Access to Document

10.1145/3332167.3357104

Cite this

Kim, T., Kim, S., Choi, J., Lee, Y., & Lee, B. (2019). Say and find it: A multimodal wearable interface for people with visual impairment. In UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology (pp. 27-29). (UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology). Association for Computing Machinery, Inc. https://doi.org/10.1145/3332167.3357104

Kim, Taeyong ; Kim, Sanghong ; Choi, Joonhee et al. / Say and find it : A multimodal wearable interface for people with visual impairment. UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, 2019. pp. 27-29 (UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology).

@inproceedings{fba187d4aecb4c78ab5ed89bda5a8157,

title = "Say and find it: A multimodal wearable interface for people with visual impairment",

abstract = "Recent advances in computer vision and natural language processing using deep neural networks (DNNs) have enabled rich and intuitive multimodal interfaces. However, research on intelligent assistance systems for persons with visual impairment has not been well explored. In this work, we present an interactive object recognition and guidance interface based on multimodal interaction for blind and partially sighted people using an embedded mobile device. We demonstrate that the proposed solution using DNNs can effectively assist visually impaired people. We believe that this work will provide new and helpful insights for designing intelligent assistance systems in the future.",

keywords = "Assistive system, Mobile interface, Multimodal wearable interface, Visual impairment",

author = "Taeyong Kim and Sanghong Kim and Joonhee Choi and Youngsun Lee and Bowon Lee",

note = "Publisher Copyright: {\textcopyright} 2019 Copyright is held by the owner/author(s).; 32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019 ; Conference date: 20-10-2019 Through 23-10-2019",

year = "2019",

month = oct,

day = "14",

doi = "10.1145/3332167.3357104",

language = "English",

series = "UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology",

publisher = "Association for Computing Machinery, Inc",

pages = "27--29",

booktitle = "UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology",

}

Kim, T, Kim, S, Choi, J, Lee, Y & Lee, B 2019, Say and find it: A multimodal wearable interface for people with visual impairment. in UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology. UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology, Association for Computing Machinery, Inc, pp. 27-29, 32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019, New Orleans, United States, 20/10/19. https://doi.org/10.1145/3332167.3357104

Say and find it: A multimodal wearable interface for people with visual impairment. / Kim, Taeyong; Kim, Sanghong; Choi, Joonhee et al.
UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc, 2019. p. 27-29 (UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Say and find it

T2 - 32nd Annual ACM Symposium on User Interface Software and Technology, UIST 2019

AU - Kim, Taeyong

AU - Kim, Sanghong

AU - Choi, Joonhee

AU - Lee, Youngsun

AU - Lee, Bowon

PY - 2019/10/14

Y1 - 2019/10/14

N2 - Recent advances in computer vision and natural language processing using deep neural networks (DNNs) have enabled rich and intuitive multimodal interfaces. However, research on intelligent assistance systems for persons with visual impairment has not been well explored. In this work, we present an interactive object recognition and guidance interface based on multimodal interaction for blind and partially sighted people using an embedded mobile device. We demonstrate that the proposed solution using DNNs can effectively assist visually impaired people. We believe that this work will provide new and helpful insights for designing intelligent assistance systems in the future.

AB - Recent advances in computer vision and natural language processing using deep neural networks (DNNs) have enabled rich and intuitive multimodal interfaces. However, research on intelligent assistance systems for persons with visual impairment has not been well explored. In this work, we present an interactive object recognition and guidance interface based on multimodal interaction for blind and partially sighted people using an embedded mobile device. We demonstrate that the proposed solution using DNNs can effectively assist visually impaired people. We believe that this work will provide new and helpful insights for designing intelligent assistance systems in the future.

KW - Assistive system

KW - Mobile interface

KW - Multimodal wearable interface

KW - Visual impairment

UR - http://www.scopus.com/inward/record.url?scp=85074853145&partnerID=8YFLogxK

U2 - 10.1145/3332167.3357104

DO - 10.1145/3332167.3357104

M3 - Conference contribution

AN - SCOPUS:85074853145

T3 - UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology

SP - 27

EP - 29

BT - UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology

PB - Association for Computing Machinery, Inc

Y2 - 20 October 2019 through 23 October 2019

ER -

Kim T, Kim S, Choi J, Lee Y, Lee B. Say and find it: A multimodal wearable interface for people with visual impairment. In UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, Inc. 2019. p. 27-29. (UIST 2019 Adjunct - Adjunct Publication of the 32nd Annual ACM Symposium on User Interface Software and Technology). doi: 10.1145/3332167.3357104

Say and find it: A multimodal wearable interface for people with visual impairment

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this