Real-time Gaze Tracking with Head-eye Coordination for Head-mounted Displays

High-accuracy, low-latency gaze tracking is becoming one of the indispensable features in augmented reality (AR) head-mounted devices (HMDs). Researchers have proposed different approaches to predict gaze positions from eye images. However, since only the eye modality is focused, these appearance-ba...

Full description

Saved in:
Bibliographic Details
Published in2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) pp. 82 - 91
Main Authors Chen, Lingling, Li, Yingxi, Bai, Xiaowei, Wang, Xiaodong, Hu, Yongqiang, Song, Mingwu, Xie, Liang, Yan, Ye, Yin, Erwei
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.10.2022
Subjects
Online AccessGet full text
DOI10.1109/ISMAR55827.2022.00022

Cover

Abstract High-accuracy, low-latency gaze tracking is becoming one of the indispensable features in augmented reality (AR) head-mounted devices (HMDs). Researchers have proposed different approaches to predict gaze positions from eye images. However, since only the eye modality is focused, these appearance-based algorithms are still struggle to trade off the accuracy and running speed in HMDs. In this paper, we propose a lightweight multi-modal network (HE-Tracker) to regress gaze positions. By fusing head-movement features with eye features, HE-Tracker achieves comparable accuracy (3.655° in all subjects) and 27 \times speedup (48 fps in the specialized AR HMD) compared to the state-of-the-art gaze tracking algorithm. We further demonstrate that when applying our head-eye coordination strategy to other baseline models, all these models achieve at least 6.36% performance improvement without a pronounced effect on running speed. Moreover, we construct HE-Gaze, the first multi-modal dataset with eye images and head-movement data for near-eye gaze tracking. This dataset is currently made of 757,360 frames and 15 persons, providing an opportunity to foster research in multi-modal gaze tracking approaches. Our dataset is available at DOWNLOAD LINK 1 .
AbstractList High-accuracy, low-latency gaze tracking is becoming one of the indispensable features in augmented reality (AR) head-mounted devices (HMDs). Researchers have proposed different approaches to predict gaze positions from eye images. However, since only the eye modality is focused, these appearance-based algorithms are still struggle to trade off the accuracy and running speed in HMDs. In this paper, we propose a lightweight multi-modal network (HE-Tracker) to regress gaze positions. By fusing head-movement features with eye features, HE-Tracker achieves comparable accuracy (3.655° in all subjects) and 27 \times speedup (48 fps in the specialized AR HMD) compared to the state-of-the-art gaze tracking algorithm. We further demonstrate that when applying our head-eye coordination strategy to other baseline models, all these models achieve at least 6.36% performance improvement without a pronounced effect on running speed. Moreover, we construct HE-Gaze, the first multi-modal dataset with eye images and head-movement data for near-eye gaze tracking. This dataset is currently made of 757,360 frames and 15 persons, providing an opportunity to foster research in multi-modal gaze tracking approaches. Our dataset is available at DOWNLOAD LINK 1 .
Author Xie, Liang
Yan, Ye
Bai, Xiaowei
Yin, Erwei
Chen, Lingling
Wang, Xiaodong
Song, Mingwu
Hu, Yongqiang
Li, Yingxi
Author_xml – sequence: 1
  givenname: Lingling
  surname: Chen
  fullname: Chen, Lingling
  organization: Hebei University of Technology,School of Artificial Intelligence and Data Science
– sequence: 2
  givenname: Yingxi
  surname: Li
  fullname: Li, Yingxi
  organization: Hebei University of Technology,School of Artificial Intelligence and Data Science
– sequence: 3
  givenname: Xiaowei
  surname: Bai
  fullname: Bai, Xiaowei
  email: xwbai@ipp.ac.cn
  organization: Defense Innovation Institute,Academy of Military Sciences,Beijing,China
– sequence: 4
  givenname: Xiaodong
  surname: Wang
  fullname: Wang, Xiaodong
  organization: Tianjin Artificial Intelligence Innovation Center,Tianjin,China
– sequence: 5
  givenname: Yongqiang
  surname: Hu
  fullname: Hu, Yongqiang
  organization: Tianjin Artificial Intelligence Innovation Center,Tianjin,China
– sequence: 6
  givenname: Mingwu
  surname: Song
  fullname: Song, Mingwu
  organization: Tianjin Artificial Intelligence Innovation Center,Tianjin,China
– sequence: 7
  givenname: Liang
  surname: Xie
  fullname: Xie, Liang
  organization: Defense Innovation Institute,Academy of Military Sciences,Beijing,China
– sequence: 8
  givenname: Ye
  surname: Yan
  fullname: Yan, Ye
  organization: Defense Innovation Institute,Academy of Military Sciences,Beijing,China
– sequence: 9
  givenname: Erwei
  surname: Yin
  fullname: Yin, Erwei
  organization: Defense Innovation Institute,Academy of Military Sciences,Beijing,China
BookMark eNotjMtKw0AUQEfQha1-gQjzA4kzd3InmWWJ2hYiQq3rcpO50cE8ShKR-PUW6uacxYGzEJdd37EQ91rFWiv3sH17We0QM0hjUACxUideiIW2FhM0gOm1KHZMTTSFluWaflnuB6q-Qvchf8L0KTdMPuKZZd73gw8dTaHvZN0P59L2393EXj6G8djQPN6Iq5qakW__vRTvz0_7fBMVr-ttviqiAMpMkUF0nFmoNYL2JgXEhJS1leW0NKUi0mw91pQhJ5ahzJz2AFllMPPekFmKu_M3MPPhOISWhvngnMMkAfMHkp9KTQ
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ISMAR55827.2022.00022
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 1665453257
9781665453257
EndPage 91
ExternalDocumentID 9995442
Genre orig-research
GrantInformation_xml – fundername: National Natural Science Foundation of China
  funderid: 10.13039/501100001809
– fundername: National Science Foundation
  funderid: 10.13039/100000001
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i203t-3559e862f1521d372554a066c6e7b3b0aa1e6d5fa85e46e2b891d228c358dd3a3
IEDL.DBID RIE
IngestDate Thu Jan 18 11:15:05 EST 2024
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-3559e862f1521d372554a066c6e7b3b0aa1e6d5fa85e46e2b891d228c358dd3a3
PageCount 10
ParticipantIDs ieee_primary_9995442
PublicationCentury 2000
PublicationDate 2022-Oct.
PublicationDateYYYYMMDD 2022-10-01
PublicationDate_xml – month: 10
  year: 2022
  text: 2022-Oct.
PublicationDecade 2020
PublicationTitle 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
PublicationTitleAbbrev ISMAR
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
Score 2.2890763
Snippet High-accuracy, low-latency gaze tracking is becoming one of the indispensable features in augmented reality (AR) head-mounted devices (HMDs). Researchers have...
SourceID ieee
SourceType Publisher
StartPage 82
SubjectTerms Artificial intelligence
Augmented reality
Computer vision
Computer vision problems
Computing methodologies
Gaze tracking
Head
Human computer interaction (HCI)
Human-centered computing
Interaction paradigms
Performance evaluation
Prediction algorithms
Predictive models
Rendering (computer graphics)
Resists
Title Real-time Gaze Tracking with Head-eye Coordination for Head-mounted Displays
URI https://ieeexplore.ieee.org/document/9995442
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1NTwIxEG2Qkyc1YPxODx4t7PZrt0eDEjRiDErCjfRjmhAVCCwH_PW2u4jGePC26R626ST73pvOzEPo0mUy0GThicwtEO40JSYLKkUpJw31DNLSY6n_KHtDfj8Soxq62vbCAEBZfAat-Fje5buZXcVUWVvF4WU8_HB3skxVvVqbppw0Ue275_71QIicZkH20XIOZ7TE_WGaUmJGdw_1v75WlYq8tlaFadmPX4MY_7udfdT87s7DT1vcOUA1mDbQwyAwPhKd4nGs38EBg2zMguOYaMW9EEkCa8CdWRCbkyoDiANfrd68R8MIcPhmspy_6fWyiYbd25dOj2ycEsiEJqwggTQoCNrERzR2LAs6getAJqyEzDCTaJ2CdMLrXACXQE2uUkdpbpnInWOaHaL6dDaFI4QFWB5vXLyRhifW56mPxn2JZtwLL9Jj1IgnMZ5XwzDGm0M4-Xv5FO3GWFTVb2eoXixWcB5QvDAXZfg-AZzZnZ8
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3PT8IwFG4IHvSkBoy_7cGjhW39se5oUALKiEFIuJG1fU2ICkTGAf962w3RGA-etmyHLX1Jv-97fe99CF2bWDiazC0RUgNhJouIip1KSRIjVGQphIXHUtoXnRF7GPNxBd1se2EAoCg-g4a_Lc7yzVyvfKqsmfjhZcxtuDvuIuOyW2vTlhMGSbP7nN4OOJdR7IRfVEzi9Ka4P2xTCtRo76P063tlschLY5Wrhv74NYrxvz90gOrf_Xn4aYs8h6gCsxrqDRznI94rHvsKHuxQSPs8OPapVtxxsSSwBtyaO7k5LXOA2DHW8s2bt4wAg--my8Vrtl7W0ah9P2x1yMYrgUyjgObE0YYEnDqxHo8NjZ1SYJmjE1pArKgKsiwEYbjNJAcmIFIyCU0USU25NIZm9AhVZ_MZHCPMQTN_5mKVUCzQVobWW_cFGWWWWx6eoJpficmiHIcx2SzC6d-Pr9BuZ5j2Jr1u__EM7fm4lLVw56iav6_gwmF6ri6LUH4CFrmg8A
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2022+IEEE+International+Symposium+on+Mixed+and+Augmented+Reality+%28ISMAR%29&rft.atitle=Real-time+Gaze+Tracking+with+Head-eye+Coordination+for+Head-mounted+Displays&rft.au=Chen%2C+Lingling&rft.au=Li%2C+Yingxi&rft.au=Bai%2C+Xiaowei&rft.au=Wang%2C+Xiaodong&rft.date=2022-10-01&rft.pub=IEEE&rft.spage=82&rft.epage=91&rft_id=info:doi/10.1109%2FISMAR55827.2022.00022&rft.externalDocID=9995442