多尺度视觉语义增强的多模态命名实体识别方法

为解决多模态命名实体识别(Multimodal named entity recognition,MNER)方法研究中存在的图像特征语义缺失和多模态表示语义约束较弱等问题,提出多尺度视觉语义增强的多模态命名实体识别方法(Multi-scale visual semant-ic enhancement for multimodal named entity recognition method,MSVSE).该方法提取多种视觉特征用于补全图像语义,挖掘文本特征与多种视觉特征间的语义交互关系,生成多尺度视觉语义特征并进行融合,得到多尺度视觉语义增强的多模态文本表示;使用视觉实体分类器对多尺度视觉语...

Full description

Saved in:
Bibliographic Details
Published in自动化学报 Vol. 50; no. 6; pp. 1234 - 1245
Main Authors 王海荣, 徐玺, 王彤, 陈芳萍
Format Journal Article
LanguageChinese
Published 北方民族大学计算机科学与工程学院 银川 750021 01.06.2024
北方民族大学图像图形智能处理国家民委重点实验室 银川 750021%北方民族大学计算机科学与工程学院 银川 750021
Subjects
多模态命名实体识别
multi-task learning
multimodal fusion
Transformer
Multimodal named entity recognition(MNER)
多模态融合
多任务学习
Online AccessGet full text
ISSN0254-4156
DOI10.16383/j.aas.c230573

Cover

Abstract 为解决多模态命名实体识别(Multimodal named entity recognition,MNER)方法研究中存在的图像特征语义缺失和多模态表示语义约束较弱等问题,提出多尺度视觉语义增强的多模态命名实体识别方法(Multi-scale visual semant-ic enhancement for multimodal named entity recognition method,MSVSE).该方法提取多种视觉特征用于补全图像语义,挖掘文本特征与多种视觉特征间的语义交互关系,生成多尺度视觉语义特征并进行融合,得到多尺度视觉语义增强的多模态文本表示;使用视觉实体分类器对多尺度视觉语义特征解码,实现视觉特征的语义一致性约束;调用多任务标签解码器挖掘多模态文本表示和文本特征的细粒度语义,通过联合解码解决语义偏差问题,从而进一步提高命名实体识别准确度.为验证该方法的有效性,在Twitter-2015和Twitter-2017数据集上进行实验,并与其他10种方法进行对比,该方法的平均F1值得到提升.
AbstractList 为解决多模态命名实体识别(Multimodal named entity recognition,MNER)方法研究中存在的图像特征语义缺失和多模态表示语义约束较弱等问题,提出多尺度视觉语义增强的多模态命名实体识别方法(Multi-scale visual semant-ic enhancement for multimodal named entity recognition method,MSVSE).该方法提取多种视觉特征用于补全图像语义,挖掘文本特征与多种视觉特征间的语义交互关系,生成多尺度视觉语义特征并进行融合,得到多尺度视觉语义增强的多模态文本表示;使用视觉实体分类器对多尺度视觉语义特征解码,实现视觉特征的语义一致性约束;调用多任务标签解码器挖掘多模态文本表示和文本特征的细粒度语义,通过联合解码解决语义偏差问题,从而进一步提高命名实体识别准确度.为验证该方法的有效性,在Twitter-2015和Twitter-2017数据集上进行实验,并与其他10种方法进行对比,该方法的平均F1值得到提升.
Abstract_FL To address the issues of semantic loss in image features and weak semantic constraints in multimodal representations encountered in the research of multimodal named entity recognition(MNER)methods,multi-scale visual semantic enhancement for multimodal named entity recognition method(MSVSE)is proposed.After supple-menting image semantics by extracting multiple visual features,the semantic interaction and feature fusion between text features and various visual features are explored through a multimodal feature fusion module.This process out-puts multi-scale visual semantic-enhanced multimodal text representations.The visual entity classifier is used to de-code multi-scale visual semantic features to learn the semantic consistency between various visual features.The multi-task decoder is invoked to mine the fine-grained semantic representation in multimodal text repre-sentation and text features,and carry out joint decoding to solve the semantic bias problem,thereby further improving the accuracy of named entity recognition.To verify the effectiveness of the method,experiments were carried out on Twitter-2015 and Twitter-2017 respectively,and compared with other 10 methods.The average F1 values of the MSVSE on the two datasets have increased.
Author 王海荣
徐玺
陈芳萍
王彤
AuthorAffiliation 北方民族大学计算机科学与工程学院 银川 750021;北方民族大学图像图形智能处理国家民委重点实验室 银川 750021%北方民族大学计算机科学与工程学院 银川 750021
AuthorAffiliation_xml – name: 北方民族大学计算机科学与工程学院 银川 750021;北方民族大学图像图形智能处理国家民委重点实验室 银川 750021%北方民族大学计算机科学与工程学院 银川 750021
Author_FL WANG Hai-Rong
XU Xi
CHEN Fang-Ping
WANG Tong
Author_FL_xml – sequence: 1
  fullname: WANG Hai-Rong
– sequence: 2
  fullname: XU Xi
– sequence: 3
  fullname: WANG Tong
– sequence: 4
  fullname: CHEN Fang-Ping
Author_xml – sequence: 1
  fullname: 王海荣
– sequence: 2
  fullname: 徐玺
– sequence: 3
  fullname: 王彤
– sequence: 4
  fullname: 陈芳萍
BookMark eNotjb1Kw1AYQO9QwVq7-ghuid93f9KbSaT4B4UuOpebm1y1SAoGUZwUpNhBqIgdOhQUsYJoBacG-za9MY9hQKczHDhniZTiThwRsoLgosckW2u7SiWupgxEjZVIGajgDkfhLZJqkhwFAIhAfcQyWbfPQ_uZ2vQlH3fzcS-fvM-nPfs0st_pz_C6sNnrY3Z5Ze9mtn9rP0bz2X0-6dqbt2wwzb4elsmCUcdJVP1nhexvbe7Vd5xGc3u3vtFwEgRkjjIiYqA5SiVYiEZQqThXNYEhQ6klCqN1EHjGi6JAhRiC9A1yKUEbXxnDKmT1r3umYqPig1a7c3oSF8fWRXh4HlCgHLzixH4BjWRjVw
ContentType Journal Article
Copyright Copyright © Wanfang Data Co. Ltd. All Rights Reserved.
Copyright_xml – notice: Copyright © Wanfang Data Co. Ltd. All Rights Reserved.
DBID 2B.
4A8
92I
93N
PSX
TCJ
DOI 10.16383/j.aas.c230573
DatabaseName Wanfang Data Journals - Hong Kong
WANFANG Data Centre
Wanfang Data Journals
万方数据期刊 - 香港版
China Online Journals (COJ)
China Online Journals (COJ)
DatabaseTitleList
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
DocumentTitle_FL Multi-scale Visual Semantic Enhancement for Multimodal Named Entity Recognition Method
EndPage 1245
ExternalDocumentID zdhxb202406013
GroupedDBID --K
-0Y
.~1
0R~
1B1
1~.
1~5
2B.
4.4
457
4A8
4G.
5GY
5VS
5XA
5XJ
7-5
71M
8P~
92H
92I
93N
AAIKJ
AALRI
AAQFI
AAXUO
ABJNI
ABWVN
ACGFS
ACRPL
ADEZE
ADNMO
ADTZH
AECPX
AEKER
AFTJW
AGHFR
AGYEJ
AITUG
ALMA_UNASSIGNED_HOLDINGS
BLXMC
CCEZO
CS3
CUBFJ
CW9
EBS
EJD
EO8
EO9
EP2
EP3
FDB
FEDTE
FNPLU
GBLVA
HVGLF
HZ~
IHE
J1W
JJJVA
M41
MO0
N9A
O-L
O9-
OAUVE
OZT
P-8
P-9
P2P
PC.
PSX
Q38
ROL
RPZ
SDF
SDG
SES
TCJ
TGT
U1G
U5S
ID FETCH-LOGICAL-s1013-af5e30c418a53d1f528a44a751d318c815fccbb6f6eebad1d089f14880cf9aff3
ISSN 0254-4156
IngestDate Thu May 29 04:10:31 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 6
Keywords 多模态命名实体识别
multi-task learning
multimodal fusion
Transformer
Multimodal named entity recognition(MNER)
多模态融合
多任务学习
Language Chinese
LinkModel OpenURL
MergedId FETCHMERGED-LOGICAL-s1013-af5e30c418a53d1f528a44a751d318c815fccbb6f6eebad1d089f14880cf9aff3
PageCount 12
ParticipantIDs wanfang_journals_zdhxb202406013
PublicationCentury 2000
PublicationDate 2024-06-01
PublicationDateYYYYMMDD 2024-06-01
PublicationDate_xml – month: 06
  year: 2024
  text: 2024-06-01
  day: 01
PublicationDecade 2020
PublicationTitle 自动化学报
PublicationTitle_FL Acta Automatica Sinica
PublicationYear 2024
Publisher 北方民族大学计算机科学与工程学院 银川 750021
北方民族大学图像图形智能处理国家民委重点实验室 银川 750021%北方民族大学计算机科学与工程学院 银川 750021
Publisher_xml – name: 北方民族大学图像图形智能处理国家民委重点实验室 银川 750021%北方民族大学计算机科学与工程学院 银川 750021
– name: 北方民族大学计算机科学与工程学院 银川 750021
SSID ssib001102911
ssib006576350
ssib051375349
ssib007293330
ssj0059721
ssib007290157
ssib023646446
ssib005904210
Score 2.480551
Snippet 为解决多模态命名实体识别(Multimodal named entity recognition,MNER)方法研究中存在的图像特征语义缺失和多模态表示语义约束较弱等问题,提出多尺度视觉语义增强的多模态命名实体识别方法(Multi-scale visual semant-ic enhancement for...
SourceID wanfang
SourceType Aggregation Database
StartPage 1234
Title 多尺度视觉语义增强的多模态命名实体识别方法
URI https://d.wanfangdata.com.cn/periodical/zdhxb202406013
Volume 50
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnR3LahUxNLS3G12IT3zbhVmV6sxNMpOsJHPv1CLqqpXuyjzt6gq2BelKQYpdCIrookJBESuIVnDVYv-mc72f4TmZ3E4qBR-b4Uxyzsl55HGSmSSEXGXQ-BLmM_xTLYcJSiigzUl4MOmJLOG5ZLh3-M7dYHqW35oTcyOjb52_lpaX0mvZyqH7Sv7Hq5AGfsVdsv_g2X2mkAAw-Bee4GF4_pWPaSyo5lRpBCKPRjWgqQ5oLKkOqdwHlAGmqO7SmNNImRQgb1MVG6qOIQ-Rm-QO54BqIPQRkB4EnpilfBp1DQApBtCx4cMxXTFbFpYuqAQ4QnIVYLkARIzWd14Ow2LElyHVRn6psUQEOkiCzLtGo8BkiWEdQWFlTKXhHYEGoWEDuKxBgeQYpaxxm1UJh1igzJo3OYoqhVIjN43CAmAUdZdI2rz5lctU6qHE4UFVPTQWpkB5U9as4I9GK4m2Q_uGxog1-dAZgKl8BxnsK1Hu2KirhcGRVg2LY8TX7QkDMRoFFlt1JyBw8-qt4rbzh4n7JE6u3ZGqPqLXtkh32IHwgzshDMRs4tDhETpbZsbHJIF6D7NPUd8j89uR4yv5wqMUDYmH9bBRMtYOITBtkbGb0e17uom2IThVzvAgFIwATjQZCDztsHkP8Zu985Ed3hlrZq94lUHgrE4In8FcGlcH6sBK4EFTZsnUmsaewYpKXT-gktmA1yuT3n0nVpw5To7ZSd64rlvsCTKysnCSHHWO_jxFblQf1qtvO9XOx8Hm6mBzbbD1ZW97rXq_Uf3Y-bn-FHL7n971Hz-pXu5WL55XXzf2dl8NtlarZ5_7b7b731-fJrNT8UxnetJeZjK56OMNKkkpCuZl3JeJYLlfirZMOE9C4ecwrGbSF2WWpWlQBkWRJrmfe1KVPg6vWamSsmRnSKv3oFecJeNlGrKSyyIRieSZlyqehmWGx3RBh1xk3jlyxao_bzurxfmDLj3_R4wL5EjTki6S1tLD5eIShN9L6WVbDX4BFRKjsg
linkProvider Elsevier
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=%E5%A4%9A%E5%B0%BA%E5%BA%A6%E8%A7%86%E8%A7%89%E8%AF%AD%E4%B9%89%E5%A2%9E%E5%BC%BA%E7%9A%84%E5%A4%9A%E6%A8%A1%E6%80%81%E5%91%BD%E5%90%8D%E5%AE%9E%E4%BD%93%E8%AF%86%E5%88%AB%E6%96%B9%E6%B3%95&rft.jtitle=%E8%87%AA%E5%8A%A8%E5%8C%96%E5%AD%A6%E6%8A%A5&rft.au=%E7%8E%8B%E6%B5%B7%E8%8D%A3&rft.au=%E5%BE%90%E7%8E%BA&rft.au=%E7%8E%8B%E5%BD%A4&rft.au=%E9%99%88%E8%8A%B3%E8%90%8D&rft.date=2024-06-01&rft.pub=%E5%8C%97%E6%96%B9%E6%B0%91%E6%97%8F%E5%A4%A7%E5%AD%A6%E8%AE%A1%E7%AE%97%E6%9C%BA%E7%A7%91%E5%AD%A6%E4%B8%8E%E5%B7%A5%E7%A8%8B%E5%AD%A6%E9%99%A2+%E9%93%B6%E5%B7%9D+750021&rft.issn=0254-4156&rft.volume=50&rft.issue=6&rft.spage=1234&rft.epage=1245&rft_id=info:doi/10.16383%2Fj.aas.c230573&rft.externalDocID=zdhxb202406013
thumbnail_s http://utb.summon.serialssolutions.com/2.0.0/image/custom?url=http%3A%2F%2Fwww.wanfangdata.com.cn%2Fimages%2FPeriodicalImages%2Fzdhxb%2Fzdhxb.jpg