R2Human: Real-Time 3D Human Appearance Rendering from a Single Image
Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R 2 Human, the first approach for real-time i...
Saved in:
Published in | Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR pp. 1187 - 1196 |
---|---|
Main Authors | , , , |
Format | Conference Proceeding |
Language | English |
Published |
IEEE
21.10.2024
|
Subjects | |
Online Access | Get full text |
ISSN | 2473-0726 |
DOI | 10.1109/ISMAR62088.2024.00135 |
Cover
Abstract | Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R 2 Human, the first approach for real-time inference and rendering of photorealistic 3D human appearance from a single image. The core of our approach is to combine the strengths of implicit texture fields and explicit neural rendering with our novel representation, namely Z-map. Based on this, we present an end-to-end network that performs high-fidelity color reconstruction of visible areas and provides reliable color inference for occluded regions. To further enhance the 3D perception ability of our network, we leverage the Fourier occupancy field as a prior for generating the texture field and providing a sampling surface in the rendering stage. We also propose a consistency loss and a spatial fusion strategy to ensure the multi-view coherence. Experimental results show that our method outperforms the state-of-the-art methods on both synthetic data and challenging real-world images, in real-time. The project page can be found at http://cic.tju. edu.cn/faculty/likun/projects/R2Human. |
---|---|
AbstractList | Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either rely on multi-camera setups or are constrained to offline operations. In this paper, we propose R 2 Human, the first approach for real-time inference and rendering of photorealistic 3D human appearance from a single image. The core of our approach is to combine the strengths of implicit texture fields and explicit neural rendering with our novel representation, namely Z-map. Based on this, we present an end-to-end network that performs high-fidelity color reconstruction of visible areas and provides reliable color inference for occluded regions. To further enhance the 3D perception ability of our network, we leverage the Fourier occupancy field as a prior for generating the texture field and providing a sampling surface in the rendering stage. We also propose a consistency loss and a spatial fusion strategy to ensure the multi-view coherence. Experimental results show that our method outperforms the state-of-the-art methods on both synthetic data and challenging real-world images, in real-time. The project page can be found at http://cic.tju. edu.cn/faculty/likun/projects/R2Human. |
Author | Lai, Yu-Kun Li, Kun Feng, Qiao Yang, Yuanwang |
Author_xml | – sequence: 1 givenname: Yuanwang surname: Yang fullname: Yang, Yuanwang organization: Tianjin University,China – sequence: 2 givenname: Qiao surname: Feng fullname: Feng, Qiao organization: Tianjin University,China – sequence: 3 givenname: Yu-Kun surname: Lai fullname: Lai, Yu-Kun organization: Cardiff University,United Kingdom – sequence: 4 givenname: Kun surname: Li fullname: Li, Kun organization: Tianjin University,China |
BookMark | eNotj91Kw0AUhFdRsK19A4V9gdSz_7velVZtoSK09bqcbM6WSLMNiV749gb1aob5hoEZs6t8zsTYvYCZEBAe1rvX-dZK8H4mQeoZgFDmgk2DC14pYYTVzlyykdROFeCkvWHjvv8AMNIbP2LLrVx9NZgf-ZbwVOzrhrha8t-Mz9uWsMMcaaC5oq7OR566c8OR7wZ_Ir5u8Ei37Drhqafpv07Y-_PTfrEqNm8v68V8U9QC7GdRoq1cGU1IWhgPSTqRKosyClEF7agMw42hgkiximBQeR19UqlEBzZGNWF3f7s1ER3arm6w-z4IcNZoG9QPPOlMvA |
CODEN | IEEPAD |
ContentType | Conference Proceeding |
DBID | 6IE 6IL CBEJK RIE RIL |
DOI | 10.1109/ISMAR62088.2024.00135 |
DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Xplore IEEE Proceedings Order Plans (POP All) 1998-Present |
DatabaseTitleList | |
Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore Digital Library (LUT) url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Computer Science |
EISBN | 9798331516475 |
EISSN | 2473-0726 |
EndPage | 1196 |
ExternalDocumentID | 10765469 |
Genre | orig-research |
GrantInformation_xml | – fundername: Science Fund for Distinguished Young Scholars of Tianjin funderid: 10.13039/501100019539 – fundername: National Natural Science Foundation of China funderid: 10.13039/501100001809 |
GroupedDBID | 6IE 6IF 6IH 6IK 6IL 6IN AAJGR AAWTH ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI OCL RIE RIL RNS |
ID | FETCH-LOGICAL-i106t-ba6d7bc59f41580f271fd6a2c11d947eb90886d7aaecdc05a384c8f3fba706cc3 |
IEDL.DBID | RIE |
IngestDate | Wed Mar 19 05:40:57 EDT 2025 |
IsPeerReviewed | false |
IsScholarly | true |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-i106t-ba6d7bc59f41580f271fd6a2c11d947eb90886d7aaecdc05a384c8f3fba706cc3 |
PageCount | 10 |
ParticipantIDs | ieee_primary_10765469 |
PublicationCentury | 2000 |
PublicationDate | 2024-Oct.-21 |
PublicationDateYYYYMMDD | 2024-10-21 |
PublicationDate_xml | – month: 10 year: 2024 text: 2024-Oct.-21 day: 21 |
PublicationDecade | 2020 |
PublicationTitle | Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR |
PublicationTitleAbbrev | ISMAR |
PublicationYear | 2024 |
Publisher | IEEE |
Publisher_xml | – name: IEEE |
SSID | ssj0052858 |
Score | 2.2751184 |
Snippet | Rendering 3D human appearance from a single image in real-time is crucial for achieving holographic communication and immersive VR/AR. Existing methods either... |
SourceID | ieee |
SourceType | Publisher |
StartPage | 1187 |
SubjectTerms | 3D human appearance Coherence Image color analysis Image reconstruction real-time Real-time systems Reliability rendering Rendering (computer graphics) single image Surface reconstruction Surface texture Synthetic data Three-dimensional displays |
Title | R2Human: Real-Time 3D Human Appearance Rendering from a Single Image |
URI | https://ieeexplore.ieee.org/document/10765469 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjZ3PS8MwFMeD7uRp_pj4mxy8ZqZpmh_exDk2YUM2B7uN_ARRN5Hu4l9vXteqCIK3kJa2JHn5Jun7vIfQpRVRexUzYiPzhJtgiWEmJ9R56ThzJnIAnEdjMZjx-3kxr2H1ioUJIVTOZ6ELxepfvl-5NRyVJQuXwN7obbQtpd7AWs20WzBVqBrRyai-Gk5HNxPBkhGlTSCDENkZpHT7kUKlUpB-G42bd28cR56769J23cevsIz__rhd1PmG9fDDlwztoa2w3EftJlsDro33APUmrDqxv8aTtDgkwH7gvIerOpwWo2nIwwhIV4F3SY_CgJ5gg6ep_BLw8DVNPR0069893g5InUOBPKXNXkmsEV5aV-iYlFrRyGQWvTDMZZnXXAYLfk7pFmOC844WJlfcqZhHayQVzuWHqLVcLcMRwkKpKHhwRkfDpVFaBm6tdxCRLgRKj1EHWmXxtgmTsWga5OSP-lO0Az0DQsCyM9Qq39fhPCl8aS-qnv0ETx2meA |
linkProvider | IEEE |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjZ1LSwMxEIAHrQc91UfFtzl4Td3NZpOsN7GWVtsifUBvJU8oaiuyvfjrTba7KoLgLWSXZEkymUx2vhmAK8VcZoSLsXLEYCqtwpLIBEfacE2Jlo4GwLk_YJ0JfZim0xJWL1gYa23hfGaboVj8yzdLvQpXZV7CeWBvsk3YSr1Zwde4VrXxpkSkooR04ii77o76t0NGvBh5M5CEINlxSOr2I4lKoUPadRhUva9dR56bq1w19cevwIz__rxdaHzjeujpSxHtwYZd7EO9yteASvE9gNaQFHf2N2joj4c40B8oaaGiDvnjqF_0YQ34p4F48U2hAJ8giUa-_GJR99VvPg2YtO_Hdx1cZlHAc2_u5VhJZrjSaea8rhaRIzx2hkmi49hklFsVPJ38K1JabXSUykRQLVzilOQR0zo5hNpiubBHgJgQjlGrZeYk5VJk3FKljA4x6ayNomNohFGZva0DZcyqATn5o_4Stjvjfm_W6w4eT2EnzFJQCyQ-g1r-vrLnXt_n6qKY5U9A_qnJ |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+-+International+Symposium+on+Mixed+and+Augmented+Reality%2C+ISMAR&rft.atitle=R2Human%3A+Real-Time+3D+Human+Appearance+Rendering+from+a+Single+Image&rft.au=Yang%2C+Yuanwang&rft.au=Feng%2C+Qiao&rft.au=Lai%2C+Yu-Kun&rft.au=Li%2C+Kun&rft.date=2024-10-21&rft.pub=IEEE&rft.eissn=2473-0726&rft.spage=1187&rft.epage=1196&rft_id=info:doi/10.1109%2FISMAR62088.2024.00135&rft.externalDocID=10765469 |