ReCog: Supporting Blind People in Recognizing Personal Objects

Dragan Ahmetovic; Daisuke Sato; Uran Oh; Tatsuya Ishihara; Kris Kitani; Chieko Asakawa

doi:10.1145/3313831.3376143

ReCog: Supporting Blind People in Recognizing Personal Objects

Dragan Ahmetovic, Daisuke Sato, Uran Oh, Tatsuya Ishihara, Kris Kitani, Chieko Asakawa

Department of Computer Science & Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

38 Scopus citations

Abstract

We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them. We report a two-session study with 10 blind participants using ReCog for object training and recognition, with and without guidance. We show that ReCog enables blind users to train and recognize their personal objects, and that camera-aiming guidance helps novice users to increase their confidence, achieve better accuracy, and learn strategies to capture better photos.

Original language	English
Title of host publication	CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450367080
DOIs	https://doi.org/10.1145/3313831.3376143
State	Published - 21 Apr 2020
Event	2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020 - Honolulu, United States Duration: 25 Apr 2020 → 30 Apr 2020

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020
Country/Territory	United States
City	Honolulu
Period	25/04/20 → 30/04/20

Bibliographical note

Funding Information:
We would like to thank all the participants who took part in our user study. This work was sponsored in part by Shimizu Corporation and Uptake (Carnegie Mellon University Machine Learning for Social Good fund).

Publisher Copyright:
© 2020 ACM.

Keywords

object recognition
photography guidance
visual impairment

Access to Document

10.1145/3313831.3376143

Cite this

Ahmetovic, D., Sato, D., Oh, U., Ishihara, T., Kitani, K., & Asakawa, C. (2020). ReCog: Supporting Blind People in Recognizing Personal Objects. In CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems Article 3376143 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3313831.3376143

@inproceedings{a8b5ded6e1644d97b825670efa19003a,

title = "ReCog: Supporting Blind People in Recognizing Personal Objects",

abstract = "We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them. We report a two-session study with 10 blind participants using ReCog for object training and recognition, with and without guidance. We show that ReCog enables blind users to train and recognize their personal objects, and that camera-aiming guidance helps novice users to increase their confidence, achieve better accuracy, and learn strategies to capture better photos.",

keywords = "object recognition, photography guidance, visual impairment",

author = "Dragan Ahmetovic and Daisuke Sato and Uran Oh and Tatsuya Ishihara and Kris Kitani and Chieko Asakawa",

note = "Funding Information: We would like to thank all the participants who took part in our user study. This work was sponsored in part by Shimizu Corporation and Uptake (Carnegie Mellon University Machine Learning for Social Good fund). Publisher Copyright: {\textcopyright} 2020 ACM.; 2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020 ; Conference date: 25-04-2020 Through 30-04-2020",

year = "2020",

month = apr,

day = "21",

doi = "10.1145/3313831.3376143",

language = "English",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

booktitle = "CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems",

}

Ahmetovic, D, Sato, D, Oh, U, Ishihara, T, Kitani, K & Asakawa, C 2020, ReCog: Supporting Blind People in Recognizing Personal Objects. in CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems., 3376143, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020, Honolulu, United States, 25/04/20. https://doi.org/10.1145/3313831.3376143

ReCog: Supporting Blind People in Recognizing Personal Objects. / Ahmetovic, Dragan; Sato, Daisuke; Oh, Uran et al.
CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2020. 3376143 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - ReCog

T2 - 2020 ACM CHI Conference on Human Factors in Computing Systems, CHI 2020

AU - Ahmetovic, Dragan

AU - Sato, Daisuke

AU - Oh, Uran

AU - Ishihara, Tatsuya

AU - Kitani, Kris

AU - Asakawa, Chieko

N1 - Funding Information: We would like to thank all the participants who took part in our user study. This work was sponsored in part by Shimizu Corporation and Uptake (Carnegie Mellon University Machine Learning for Social Good fund). Publisher Copyright: © 2020 ACM.

PY - 2020/4/21

Y1 - 2020/4/21

N2 - We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them. We report a two-session study with 10 blind participants using ReCog for object training and recognition, with and without guidance. We show that ReCog enables blind users to train and recognize their personal objects, and that camera-aiming guidance helps novice users to increase their confidence, achieve better accuracy, and learn strategies to capture better photos.

AB - We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them. We report a two-session study with 10 blind participants using ReCog for object training and recognition, with and without guidance. We show that ReCog enables blind users to train and recognize their personal objects, and that camera-aiming guidance helps novice users to increase their confidence, achieve better accuracy, and learn strategies to capture better photos.

KW - object recognition

KW - photography guidance

KW - visual impairment

UR - http://www.scopus.com/inward/record.url?scp=85091302319&partnerID=8YFLogxK

U2 - 10.1145/3313831.3376143

DO - 10.1145/3313831.3376143

M3 - Conference contribution

AN - SCOPUS:85091302319

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI 2020 - Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery

Y2 - 25 April 2020 through 30 April 2020

ER -

ReCog: Supporting Blind People in Recognizing Personal Objects

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this