Federated learning-driven collaborative recommendation system for multi-modal art analysis and enhanced recommendations

• Published in: PeerJ. Computer science Vol. 10; p. e2405
• Main Authors: Gong, Bei, Mahsan, Ida Puteri, Xiao, Junhua
• Format: Journal Article
• Language: English
• Published: United States PeerJ. Ltd 27.11.2024; PeerJ, Inc; PeerJ Inc
• Subjects: Art similarity search; Art works; Artificial intelligence; Artificial neural networks; Artists; Collaboration; Data integration; Data mining; Data privacy; Data security; Federated learning; Machine learning; Methods; Multimodal data fusion; Recommender systems; Similarity; Technology assessment; China; Data privacy; Federated learning; Artificial intelligence; Art similarity search; Multimodal data fusion
• ISSN: 2376-5992; 2376-5992
• DOI: 10.7717/peerj-cs.2405

With the rapid development of artificial intelligence technology, recommendation systems have been widely applied in various fields. However, in the art field, art similarity search and recommendation systems face unique challenges, namely data privacy and copyright protection issues. To address these problems, this article proposes a cross-institutional artwork similarity search and recommendation system (AI-based Collaborative Recommendation System (AICRS) framework) that combines multimodal data fusion and federated learning. This system uses pre-trained convolutional neural networks (CNN) and Bidirectional Encoder Representation from Transformers (BERT) models to extract features from image and text data. It then uses a federated learning framework to train models locally at each participating institution and aggregate parameters to optimize the global model. Experimental results show that the AICRS framework achieves a final accuracy of 92.02% on the SemArt dataset, compared to 81.52% and 83.44% for traditional CNN and Long Short-Term Memory (LSTM) models, respectively. The final loss value of the AICRS framework is 0.1284, which is better than the 0.248 and 0.188 of CNN and LSTM models. The research results of this article not only provide an effective technical solution but also offer strong support for the recommendation and protection of artworks in practice.