Object-driven figure image generation method based on diffusion model - ZHANG WEIZHONG; JIN CHENG; WANG YIBIN (2024)
The invention discloses an object-driven figure image generation method based on a diffusion model. According to the method, an image generation process is divided into three continuous stages, namely semantic scene construction, subject-scene fusion and subject enhancement. The first and third stages are separately executed by a text diffusion model (TDM) and a subject diffusion model (SDM), respectively, while the second stage is completed by a saliency adaptive noise fusion (SNF) mechanism, in each time step of generation, the SNF makes use of their respective advantages through a classifier-free response of each pre-training model, and in each time step of generation, the first stage and the second stage are completed by a saliency adaptive noise fusion (SNF) mechanism. Noise predicted by the two models is fused spatially in a self-adaptive manner in a saliency perception manner, so that cooperative generation of the two models is realized. According to the method, the problems of training imbalance and q