文本数据质量确定的方法及装置

• Format: Patent
• Language: Chinese
• Published: 27.02.2024
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

本申请提供一种文本数据质量确定的方法及装置，其中该方法包括：若确定文本数据的获取来源或对应的数据类型，不满足高质量预选规则，且文本数据的主体语言为简体中文，则确定文本数据进行第一字符预处理后的结果，作为第一文本数据；基于神经网络模型，确定第一文本数据对应的正向困惑度、反向困惑度和第一置信度；基于逻辑回归模型，确定第一文本数据对应的第二置信度；若第二置信度大于第一阈值，则确定第一文本数据属于高质量的文本数据。本申请通过对任意来源的文本数据进行第一字符预处理，对数据进行全面的清洗和过滤，提高文本数据质量，并可适用于大规模预训练模型，进而提高预训练模型的性能，具有良好的实用性。 The invention provides a text data quality determination method and device.The method comprises the steps that if it is determined that an acquisition source or a corresponding data type of text data does not meet a high-quality pre-selection rule and a subject language of the text data is simplified Chinese, a result obtained after first character preprocessing is conducted on the text data is determined, the data serve as first text data; based on a neural network model, determining a forward confusion degree, a reverse confusion degree and a first confidence degree corresponding to the first text data; determining a second confidence coefficient corresponding to the first text data based on a logistic regression model; an