Object-driven figure image generation method based on diffusion model

• Main Authors: ZHANG WEIZHONG, JIN CHENG, WANG YIBIN
• Format: Patent
• Language: Chinese; English
• Published: 19.07.2024
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; IMAGE DATA PROCESSING OR GENERATION, IN GENERAL; PHYSICS

The invention discloses an object-driven figure image generation method based on a diffusion model. According to the method, an image generation process is divided into three continuous stages, namely semantic scene construction, subject-scene fusion and subject enhancement. The first and third stages are separately executed by a text diffusion model (TDM) and a subject diffusion model (SDM), respectively, while the second stage is completed by a saliency adaptive noise fusion (SNF) mechanism, in each time step of generation, the SNF makes use of their respective advantages through a classifier-free response of each pre-training model, and in each time step of generation, the first stage and the second stage are completed by a saliency adaptive noise fusion (SNF) mechanism. Noise predicted by the two models is fused spatially in a self-adaptive manner in a saliency perception manner, so that cooperative generation of the two models is realized. According to the method, the problems of training imbalance and q