Eff-YNet: A Dual Task Network for DeepFake Detection and Segmentation
Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to identify for both humans and machines. Modern detection methods exploit various weaknesses in deepfake videos, such as visual artifacts and inconsis...
Saved in:
Published in | 2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM) pp. 1 - 8 |
---|---|
Main Authors | , , |
Format | Conference Proceeding |
Language | English |
Published |
IEEE
04.01.2021
|
Subjects | |
Online Access | Get full text |
DOI | 10.1109/IMCOM51814.2021.9377373 |
Cover
Abstract | Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to identify for both humans and machines. Modern detection methods exploit various weaknesses in deepfake videos, such as visual artifacts and inconsistent posing. In this paper, we describe a novel architecture called Eff-YNet designed to detect visual differences between altered and unaltered areas. The architecture combines an EfficientNet encoder and a U-Net with a classification branch into a model capable of both classifying and segmenting deepfake videos. The task of segmentation helps train the classifier and also produces useful segmentation masks. We also implement ResNet 3D to detect spatiotemporal inconsistencies. To test these models, we run experiments against the Deepfake Detection Challenge dataset and show improvements over baseline classification models. Furthermore, we find that an ensemble of these two approaches improves performance over a single approach alone. |
---|---|
AbstractList | Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to identify for both humans and machines. Modern detection methods exploit various weaknesses in deepfake videos, such as visual artifacts and inconsistent posing. In this paper, we describe a novel architecture called Eff-YNet designed to detect visual differences between altered and unaltered areas. The architecture combines an EfficientNet encoder and a U-Net with a classification branch into a model capable of both classifying and segmenting deepfake videos. The task of segmentation helps train the classifier and also produces useful segmentation masks. We also implement ResNet 3D to detect spatiotemporal inconsistencies. To test these models, we run experiments against the Deepfake Detection Challenge dataset and show improvements over baseline classification models. Furthermore, we find that an ensemble of these two approaches improves performance over a single approach alone. |
Author | Tjon, Eric Moh, Melody Moh, Teng-Sheng |
Author_xml | – sequence: 1 givenname: Eric surname: Tjon fullname: Tjon, Eric email: eric.tjon@sjsu.edu organization: San José State University,Department of Computer Science,San José,CA,USA – sequence: 2 givenname: Melody surname: Moh fullname: Moh, Melody email: melody.moh@sjsu.edu organization: San José State University,Department of Computer Science,San José,CA,USA – sequence: 3 givenname: Teng-Sheng surname: Moh fullname: Moh, Teng-Sheng email: teng.moh@sjsu.edu organization: San José State University,Department of Computer Science,San José,CA,USA |
BookMark | eNotj8tKw0AYhUfQha0-gQvnBRJn8mdu7kqa1kJrF9aFqzKXfySkTUo6Ir69EQsHvsO3OHAm5LrrOyTkkbOcc2aeVptquxFc8zIvWMFzA0qBgisy4VKKsgCu9S2p6xizj1dMz3RG51_2QHf23NJRfPdDS2M_0DniaWFbHEtCn5q-o7YL9A0_j9gl-yfuyE20hzPeXzgl74t6V71k6-1yVc3WWVMwSBkYJrUInjGjA4oAzjHkoURtpZNBCIPSaWfUGOm9jzGKoGUAxayWgDAlD_-7DSLuT0NztMPP_vIMfgFHWkhW |
ContentType | Conference Proceeding |
DBID | 6IE 6IL CBEJK RIE RIL |
DOI | 10.1109/IMCOM51814.2021.9377373 |
DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Xplore POP ALL IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
DatabaseTitleList | |
Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore Electronic Library url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
EISBN | 1665423188 9781665423182 |
EndPage | 8 |
ExternalDocumentID | 9377373 |
Genre | orig-research |
GroupedDBID | 6IE 6IL CBEJK RIE RIL |
ID | FETCH-LOGICAL-i203t-390685dc0098de5d3bb0e1d4e8a6b6d559e6b8b97b976cccfff5d86d370a863e3 |
IEDL.DBID | RIE |
IngestDate | Thu Jun 29 18:38:24 EDT 2023 |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-i203t-390685dc0098de5d3bb0e1d4e8a6b6d559e6b8b97b976cccfff5d86d370a863e3 |
PageCount | 8 |
ParticipantIDs | ieee_primary_9377373 |
PublicationCentury | 2000 |
PublicationDate | 2021-Jan.-4 |
PublicationDateYYYYMMDD | 2021-01-04 |
PublicationDate_xml | – month: 01 year: 2021 text: 2021-Jan.-4 day: 04 |
PublicationDecade | 2020 |
PublicationTitle | 2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM) |
PublicationTitleAbbrev | IMCOM |
PublicationYear | 2021 |
Publisher | IEEE |
Publisher_xml | – name: IEEE |
Score | 1.7906431 |
Snippet | Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to... |
SourceID | ieee |
SourceType | Publisher |
StartPage | 1 |
SubjectTerms | computer vision deep learning Deepfake detection image classification image segmentation Information integrity Information management Spatiotemporal phenomena Task analysis Three-dimensional displays U-Net Videos Visualization |
Title | Eff-YNet: A Dual Task Network for DeepFake Detection and Segmentation |
URI | https://ieeexplore.ieee.org/document/9377373 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LSwMxEA5tT55UWvFNDh7NNm12k6w36YMqbBVsoZ7KJpmIVLdFdy_-epPdtaJ4EHIYQiDJDORLJt_MIHTBLLizEQQRLFIkFDQmUgEl0teosUwpW9YhS6Z8Mg9vF9GigS63sTAAUJLPIPBi-Zdv1rrwrrKug1LBBGuiphBxFatVU7Z6NO7eJIO7JHKI5V0l_V5Qj_5RNqVEjfEuSr7mq8giq6DIVaA_fqVi_O-C9lDnOz4P32-RZx81IGuj0cha8jiF_Apf42GRvuBZ-r7C04rnjd3lFA8BNuN0BU7ISwpWhtPM4Ad4eq1DkLIOmo9Hs8GE1EUSyHOfspywmHIZGe0TgxqIjFMvhZ4JQaZcceMeDMCVVLFwjWutrbWRkdwwQVPJGbAD1MrWGRwibIS7DxotfQ79UBqqeN-yUIcAIgXN2BFqexUsN1UejGW9--O_u0_QjjdD6a4IT1ErfyvgzAF4rs5Ly30CBxqbqg |
linkProvider | IEEE |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3NS8MwFA9zHvSksonf5uDRdt3SJqk32QebrlVwg3kaTfIiMu2Gthf_epO2ThQPQg4hBJK8QH7vvfzeewhdEA3mbQTmMBIIx2de6HABnsNtjRpNhNBFHbIopsOpfzMLZjV0uY6FAYCCfAau7RZ_-Wopc-sqaxkoZYSRDbQZGKuCldFaFWmr7YWtUdS9iwKDWdZZ0mm71fwfhVMK3BjsoOhrxZIusnDzTLjy41cyxv9uaRc1vyP08P0ae_ZQDdIG6ve1dh5jyK7wNe7lyQueJO8LHJdMb2zUU9wDWA2SBZhOVpCwUpykCj_A02sVhJQ20XTQn3SHTlUmwXnueCRzSOhRHihpU4MqCJQRsAdt5QNPqKDKmAxABRchM41KKbXWgeJUEeYlnBIg-6ieLlM4QFgxoxEqyW0WfZ8rT9COJr70AVgCkpBD1LAimK_KTBjz6vRHfw-fo63hJBrPx6P49hht2yspnBf-CapnbzmcGjjPxFlxi59m4p77 |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+15th+International+Conference+on+Ubiquitous+Information+Management+and+Communication+%28IMCOM%29&rft.atitle=Eff-YNet%3A+A+Dual+Task+Network+for+DeepFake+Detection+and+Segmentation&rft.au=Tjon%2C+Eric&rft.au=Moh%2C+Melody&rft.au=Moh%2C+Teng-Sheng&rft.date=2021-01-04&rft.pub=IEEE&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FIMCOM51814.2021.9377373&rft.externalDocID=9377373 |