Eff-YNet: A Dual Task Network for DeepFake Detection and Segmentation

Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to identify for both humans and machines. Modern detection methods exploit various weaknesses in deepfake videos, such as visual artifacts and inconsis...

Full description

Saved in:
Bibliographic Details
Published in2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM) pp. 1 - 8
Main Authors Tjon, Eric, Moh, Melody, Moh, Teng-Sheng
Format Conference Proceeding
LanguageEnglish
Published IEEE 04.01.2021
Subjects
Online AccessGet full text
DOI10.1109/IMCOM51814.2021.9377373

Cover

Abstract Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to identify for both humans and machines. Modern detection methods exploit various weaknesses in deepfake videos, such as visual artifacts and inconsistent posing. In this paper, we describe a novel architecture called Eff-YNet designed to detect visual differences between altered and unaltered areas. The architecture combines an EfficientNet encoder and a U-Net with a classification branch into a model capable of both classifying and segmenting deepfake videos. The task of segmentation helps train the classifier and also produces useful segmentation masks. We also implement ResNet 3D to detect spatiotemporal inconsistencies. To test these models, we run experiments against the Deepfake Detection Challenge dataset and show improvements over baseline classification models. Furthermore, we find that an ensemble of these two approaches improves performance over a single approach alone.
AbstractList Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to identify for both humans and machines. Modern detection methods exploit various weaknesses in deepfake videos, such as visual artifacts and inconsistent posing. In this paper, we describe a novel architecture called Eff-YNet designed to detect visual differences between altered and unaltered areas. The architecture combines an EfficientNet encoder and a U-Net with a classification branch into a model capable of both classifying and segmenting deepfake videos. The task of segmentation helps train the classifier and also produces useful segmentation masks. We also implement ResNet 3D to detect spatiotemporal inconsistencies. To test these models, we run experiments against the Deepfake Detection Challenge dataset and show improvements over baseline classification models. Furthermore, we find that an ensemble of these two approaches improves performance over a single approach alone.
Author Tjon, Eric
Moh, Melody
Moh, Teng-Sheng
Author_xml – sequence: 1
  givenname: Eric
  surname: Tjon
  fullname: Tjon, Eric
  email: eric.tjon@sjsu.edu
  organization: San José State University,Department of Computer Science,San José,CA,USA
– sequence: 2
  givenname: Melody
  surname: Moh
  fullname: Moh, Melody
  email: melody.moh@sjsu.edu
  organization: San José State University,Department of Computer Science,San José,CA,USA
– sequence: 3
  givenname: Teng-Sheng
  surname: Moh
  fullname: Moh, Teng-Sheng
  email: teng.moh@sjsu.edu
  organization: San José State University,Department of Computer Science,San José,CA,USA
BookMark eNotj8tKw0AYhUfQha0-gQvnBRJn8mdu7kqa1kJrF9aFqzKXfySkTUo6Ir69EQsHvsO3OHAm5LrrOyTkkbOcc2aeVptquxFc8zIvWMFzA0qBgisy4VKKsgCu9S2p6xizj1dMz3RG51_2QHf23NJRfPdDS2M_0DniaWFbHEtCn5q-o7YL9A0_j9gl-yfuyE20hzPeXzgl74t6V71k6-1yVc3WWVMwSBkYJrUInjGjA4oAzjHkoURtpZNBCIPSaWfUGOm9jzGKoGUAxayWgDAlD_-7DSLuT0NztMPP_vIMfgFHWkhW
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/IMCOM51814.2021.9377373
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Xplore POP ALL
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Xplore Electronic Library
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 1665423188
9781665423182
EndPage 8
ExternalDocumentID 9377373
Genre orig-research
GroupedDBID 6IE
6IL
CBEJK
RIE
RIL
ID FETCH-LOGICAL-i203t-390685dc0098de5d3bb0e1d4e8a6b6d559e6b8b97b976cccfff5d86d370a863e3
IEDL.DBID RIE
IngestDate Thu Jun 29 18:38:24 EDT 2023
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i203t-390685dc0098de5d3bb0e1d4e8a6b6d559e6b8b97b976cccfff5d86d370a863e3
PageCount 8
ParticipantIDs ieee_primary_9377373
PublicationCentury 2000
PublicationDate 2021-Jan.-4
PublicationDateYYYYMMDD 2021-01-04
PublicationDate_xml – month: 01
  year: 2021
  text: 2021-Jan.-4
  day: 04
PublicationDecade 2020
PublicationTitle 2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM)
PublicationTitleAbbrev IMCOM
PublicationYear 2021
Publisher IEEE
Publisher_xml – name: IEEE
Score 1.7906431
Snippet Advances in generative models and manipulation techniques have given rise to digitally altered videos known as deepfakes. These videos are difficult to...
SourceID ieee
SourceType Publisher
StartPage 1
SubjectTerms computer vision
deep learning
Deepfake detection
image classification
image segmentation
Information integrity
Information management
Spatiotemporal phenomena
Task analysis
Three-dimensional displays
U-Net
Videos
Visualization
Title Eff-YNet: A Dual Task Network for DeepFake Detection and Segmentation
URI https://ieeexplore.ieee.org/document/9377373
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LSwMxEA5tT55UWvFNDh7NNm12k6w36YMqbBVsoZ7KJpmIVLdFdy_-epPdtaJ4EHIYQiDJDORLJt_MIHTBLLizEQQRLFIkFDQmUgEl0teosUwpW9YhS6Z8Mg9vF9GigS63sTAAUJLPIPBi-Zdv1rrwrrKug1LBBGuiphBxFatVU7Z6NO7eJIO7JHKI5V0l_V5Qj_5RNqVEjfEuSr7mq8giq6DIVaA_fqVi_O-C9lDnOz4P32-RZx81IGuj0cha8jiF_Apf42GRvuBZ-r7C04rnjd3lFA8BNuN0BU7ISwpWhtPM4Ad4eq1DkLIOmo9Hs8GE1EUSyHOfspywmHIZGe0TgxqIjFMvhZ4JQaZcceMeDMCVVLFwjWutrbWRkdwwQVPJGbAD1MrWGRwibIS7DxotfQ79UBqqeN-yUIcAIgXN2BFqexUsN1UejGW9--O_u0_QjjdD6a4IT1ErfyvgzAF4rs5Ly30CBxqbqg
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV3NS8MwFA9zHvSksonf5uDRdt3SJqk32QebrlVwg3kaTfIiMu2Gthf_epO2ThQPQg4hBJK8QH7vvfzeewhdEA3mbQTmMBIIx2de6HABnsNtjRpNhNBFHbIopsOpfzMLZjV0uY6FAYCCfAau7RZ_-Wopc-sqaxkoZYSRDbQZGKuCldFaFWmr7YWtUdS9iwKDWdZZ0mm71fwfhVMK3BjsoOhrxZIusnDzTLjy41cyxv9uaRc1vyP08P0ae_ZQDdIG6ve1dh5jyK7wNe7lyQueJO8LHJdMb2zUU9wDWA2SBZhOVpCwUpykCj_A02sVhJQ20XTQn3SHTlUmwXnueCRzSOhRHihpU4MqCJQRsAdt5QNPqKDKmAxABRchM41KKbXWgeJUEeYlnBIg-6ieLlM4QFgxoxEqyW0WfZ8rT9COJr70AVgCkpBD1LAimK_KTBjz6vRHfw-fo63hJBrPx6P49hht2yspnBf-CapnbzmcGjjPxFlxi59m4p77
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+15th+International+Conference+on+Ubiquitous+Information+Management+and+Communication+%28IMCOM%29&rft.atitle=Eff-YNet%3A+A+Dual+Task+Network+for+DeepFake+Detection+and+Segmentation&rft.au=Tjon%2C+Eric&rft.au=Moh%2C+Melody&rft.au=Moh%2C+Teng-Sheng&rft.date=2021-01-04&rft.pub=IEEE&rft.spage=1&rft.epage=8&rft_id=info:doi/10.1109%2FIMCOM51814.2021.9377373&rft.externalDocID=9377373