R2Human: Real-Time 3D Human Appearance Rendering from a Single Image

Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R 2 Human, the first approach for real-time i...

Full description

Saved in:
Bibliographic Details
Published inProceedings - International Symposium on Mixed and Augmented Reality, ISMAR pp. 1187 - 1196
Main Authors Yang, Yuanwang, Feng, Qiao, Lai, Yu-Kun, Li, Kun
Format Conference Proceeding
LanguageEnglish
Published IEEE 21.10.2024
Subjects
Online AccessGet full text
ISSN2473-0726
DOI10.1109/ISMAR62088.2024.00135

Cover

Abstract Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R 2 Human, the first approach for real-time inference and rendering of photorealistic 3D human appearance from a single image. The core of our approach is to combine the strengths of implicit texture fields and explicit neural rendering with our novel representation, namely Z-map. Based on this, we present an end-to-end network that performs high-fidelity color reconstruction of visible areas and provides reliable color inference for occluded regions. To further enhance the 3D perception ability of our network, we leverage the Fourier occupancy field as a prior for generating the texture field and providing a sampling surface in the rendering stage. We also propose a consistency loss and a spatial fusion strategy to ensure the multi-view coherence. Experimental results show that our method outperforms the state-of-the-art methods on both synthetic data and challenging real-world images, in real-time. The project page can be found at http://cic.tju. edu.cn/faculty/likun/projects/R2Human.
AbstractList Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R 2 Human, the first approach for real-time inference and rendering of photorealistic 3D human appearance from a single image. The core of our approach is to combine the strengths of implicit texture fields and explicit neural rendering with our novel representation, namely Z-map. Based on this, we present an end-to-end network that performs high-fidelity color reconstruction of visible areas and provides reliable color inference for occluded regions. To further enhance the 3D perception ability of our network, we leverage the Fourier occupancy field as a prior for generating the texture field and providing a sampling surface in the rendering stage. We also propose a consistency loss and a spatial fusion strategy to ensure the multi-view coherence. Experimental results show that our method outperforms the state-of-the-art methods on both synthetic data and challenging real-world images, in real-time. The project page can be found at http://cic.tju. edu.cn/faculty/likun/projects/R2Human.
Author Lai, Yu-Kun
Li, Kun
Feng, Qiao
Yang, Yuanwang
Author_xml – sequence: 1
  givenname: Yuanwang
  surname: Yang
  fullname: Yang, Yuanwang
  organization: Tianjin University,China
– sequence: 2
  givenname: Qiao
  surname: Feng
  fullname: Feng, Qiao
  organization: Tianjin University,China
– sequence: 3
  givenname: Yu-Kun
  surname: Lai
  fullname: Lai, Yu-Kun
  organization: Cardiff University,United Kingdom
– sequence: 4
  givenname: Kun
  surname: Li
  fullname: Li, Kun
  organization: Tianjin University,China
BookMark eNotj91Kw0AUhFdRsK19A4V9gdSz_7velVZtoSK09bqcbM6WSLMNiV749gb1aob5hoEZs6t8zsTYvYCZEBAe1rvX-dZK8H4mQeoZgFDmgk2DC14pYYTVzlyykdROFeCkvWHjvv8AMNIbP2LLrVx9NZgf-ZbwVOzrhrha8t-Mz9uWsMMcaaC5oq7OR566c8OR7wZ_Ir5u8Ei37Drhqafpv07Y-_PTfrEqNm8v68V8U9QC7GdRoq1cGU1IWhgPSTqRKosyClEF7agMw42hgkiximBQeR19UqlEBzZGNWF3f7s1ER3arm6w-z4IcNZoG9QPPOlMvA
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/ISMAR62088.2024.00135
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Xplore
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore Digital Library (LUT)
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9798331516475
EISSN 2473-0726
EndPage 1196
ExternalDocumentID 10765469
Genre orig-research
GrantInformation_xml – fundername: Science Fund for Distinguished Young Scholars of Tianjin
  funderid: 10.13039/501100019539
– fundername: National Natural Science Foundation of China
  funderid: 10.13039/501100001809
GroupedDBID 6IE
6IF
6IH
6IK
6IL
6IN
AAJGR
AAWTH
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
OCL
RIE
RIL
RNS
ID FETCH-LOGICAL-i106t-ba6d7bc59f41580f271fd6a2c11d947eb90886d7aaecdc05a384c8f3fba706cc3
IEDL.DBID RIE
IngestDate Wed Mar 19 05:40:57 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i106t-ba6d7bc59f41580f271fd6a2c11d947eb90886d7aaecdc05a384c8f3fba706cc3
PageCount 10
ParticipantIDs ieee_primary_10765469
PublicationCentury 2000
PublicationDate 2024-Oct.-21
PublicationDateYYYYMMDD 2024-10-21
PublicationDate_xml – month: 10
  year: 2024
  text: 2024-Oct.-21
  day: 21
PublicationDecade 2020
PublicationTitle Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR
PublicationTitleAbbrev ISMAR
PublicationYear 2024
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0052858
Score 2.2751184
Snippet Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either...
SourceID ieee
SourceType Publisher
StartPage 1187
SubjectTerms 3D human appearance
Coherence
Image color analysis
Image reconstruction
real-time
Real-time systems
Reliability
rendering
Rendering (computer graphics)
single image
Surface reconstruction
Surface texture
Synthetic data
Three-dimensional displays
Title R2Human: Real-Time 3D Human Appearance Rendering from a Single Image
URI https://ieeexplore.ieee.org/document/10765469
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjZ3PS8MwFMeD7uRp_pj4mxy8ZqZpmh_exDk2YUM2B7uN_ARRN5Hu4l9vXteqCIK3kJa2JHn5Jun7vIfQpRVRexUzYiPzhJtgiWEmJ9R56ThzJnIAnEdjMZjx-3kxr2H1ioUJIVTOZ6ELxepfvl-5NRyVJQuXwN7obbQtpd7AWs20WzBVqBrRyai-Gk5HNxPBkhGlTSCDENkZpHT7kUKlUpB-G42bd28cR56769J23cevsIz__rhd1PmG9fDDlwztoa2w3EftJlsDro33APUmrDqxv8aTtDgkwH7gvIerOpwWo2nIwwhIV4F3SY_CgJ5gg6ep_BLw8DVNPR0069893g5InUOBPKXNXkmsEV5aV-iYlFrRyGQWvTDMZZnXXAYLfk7pFmOC844WJlfcqZhHayQVzuWHqLVcLcMRwkKpKHhwRkfDpVFaBm6tdxCRLgRKj1EHWmXxtgmTsWga5OSP-lO0Az0DQsCyM9Qq39fhPCl8aS-qnv0ETx2meA
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjZ1LSwMxEIAHrQc91UfFtzl4Td3NZpOsN7GWVtsifUBvJU8oaiuyvfjrTba7KoLgLWSXZEkymUx2vhmAK8VcZoSLsXLEYCqtwpLIBEfacE2Jlo4GwLk_YJ0JfZim0xJWL1gYa23hfGaboVj8yzdLvQpXZV7CeWBvsk3YSr1Zwde4VrXxpkSkooR04ii77o76t0NGvBh5M5CEINlxSOr2I4lKoUPadRhUva9dR56bq1w19cevwIz__rxdaHzjeujpSxHtwYZd7EO9yteASvE9gNaQFHf2N2joj4c40B8oaaGiDvnjqF_0YQ34p4F48U2hAJ8giUa-_GJR99VvPg2YtO_Hdx1cZlHAc2_u5VhJZrjSaea8rhaRIzx2hkmi49hklFsVPJ38K1JabXSUykRQLVzilOQR0zo5hNpiubBHgJgQjlGrZeYk5VJk3FKljA4x6ayNomNohFGZva0DZcyqATn5o_4Stjvjfm_W6w4eT2EnzFJQCyQ-g1r-vrLnXt_n6qKY5U9A_qnJ
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+-+International+Symposium+on+Mixed+and+Augmented+Reality%2C+ISMAR&rft.atitle=R2Human%3A+Real-Time+3D+Human+Appearance+Rendering+from+a+Single+Image&rft.au=Yang%2C+Yuanwang&rft.au=Feng%2C+Qiao&rft.au=Lai%2C+Yu-Kun&rft.au=Li%2C+Kun&rft.date=2024-10-21&rft.pub=IEEE&rft.eissn=2473-0726&rft.spage=1187&rft.epage=1196&rft_id=info:doi/10.1109%2FISMAR62088.2024.00135&rft.externalDocID=10765469