多策略融合的俄语文本词语提取方法研究

doi:10.12339/j.issn.1673-8578.2021.03.009

摘要/Abstract

摘要：

俄语是联合国工作语言之一,是俄罗斯等多个国家的官方语言。随着“一带一路”倡议的推进和全球化进程的加快,俄语文本数据成为有关组织管理决策的重要信息来源,俄语文本挖掘也因而成为重要的管理决策支持方法。然而,俄语文本挖掘方法研究目前还远未成熟,尤其是其关键基础——俄语文本词语提取的性能较低,阻碍着俄语文本建模的准确性。因此,文章提出一种多策略融合的俄语文本词语提取方法,结合俄语词性分析、语法规则和串频统计等多种方法,自动提取包含单词和短语在内的俄语词语。在联合国平行语料库和Taiga Corpus语料库上的实验结果表明,文章提出的方法在保证高召回率的同时,达到了85%以上的高准确率,显著优于常用的n-gram方法,能够为俄语文本主题发现和文本分/聚类等文本挖掘应用提供有效的词库。

关键词: 俄语文本挖掘, 词语提取, 词性标注, 频繁词串

Abstract:

Russian is one of the working languages of the United Nations and the official language of many countries including Russia. With the advancement of the Belt and Road Initiative and the acceleration of globalization, Russian text data has become an important information resource for managerial decision-making of related organizations and Russian text mining has thus become a significant decision-making method. However, Russian text mining methods are still far away from being mature, especially the essential Russian text term extraction method, which affects the accuracy of Russian text modeling. This paper proposes a Russian text term extraction method, which combines multi strategies including Russian POS analysis, grammatical rules and string frequency statistics to automatically extract Russian words and multiword expressions. Experiments on the United Nations Parallel Corpus and the Taiga Corpus show that the proposed method achieves a high accuracy of approximate 85% which is much higher than normal recall rate, such as the n-gram method. The proposed method can be used to create lexicons for Russian text mining applications such as text topic discovery, text classification, and text clustering.

Key words: Russian text mining, term extraction, POS tag, frequent word-string

中图分类号:

唐菊香, 孙怿晖, 廖晓, 刘建国, 于娟. 多策略融合的俄语文本词语提取方法研究[J]. 中国科技术语, 2021, 23(3): 59-67.

TANG Juxiang, SUN Yihui, LIAO Xiao, LIU Jianguo, YU Juan. Extracting Terms from Russian Texts Based on Multi Strategies[J]. China Terminology, 2021, 23(3): 59-67.

图/表 8

参考文献 35

[1]	JACKENDOFF R, CYNX J. The architecture of the language faculty[J]. Quarterly Review of Biology, 1997,7(74):1-8. doi: 10.1086/394393 URL
[2]	FIROOZEH N, NAZARENKO A, ALIZON F. Keyword extraction: Issues and methods[J]. Natural Language Engineering, 2020,26(3):259-291. doi: 10.1017/S1351324919000457 URL
[3]	VILLAVICENCIO A, IDIART M. Discovering multiword expressions[J]. Natural Language Engineering, 2019,25(6):715-733. doi: 10.1017/S1351324919000494 URL
[4]	于娟, 党延忠. 结合词性分析与串频统计的词语提取方法[J]. 系统工程理论与实践, 2010,30(1):105-111.
[5]	HASAN K S, NG V. Automatic keyphrase extraction: A survey of the state of the art[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics,ACL 2014. Baltimore, Maryland, USA: ACL Press, 2014: 1262-1273.
[6]	LOUKACHEVITCH N, PARKHOMENKO E, LOUKACH-EVITCH N. Evaluating distributional features for multiword expression recognition[C]// 21st International Conference on Text, Speech, and Dialogue, TSD 2018. Brno, Czech Republic: Springer, Cham, 2018: 126-134.
[7]	李峰, 易绵竹. 面向俄文NLP的形态自动分析研究与实现[J]. 中文信息学报, 2011,25(5):68-75.
[8]	GOLDSMITH J. Unsupervised learning of the morphology of a natural language[J]. Computational linguistics, 2001,27(2):153-198. doi: 10.1162/089120101750300490 URL
[9]	ЛАПШИН С В, ЛЕБЕДЕВ И С. Метод полуавтомати-ческого формирования словаря морфологических описаний слов[J]. Научно-технический вестник информационных технологий, механики и оптики, 2012,5(81):104-107.
[10]	Yandex. MyStem [EB/OL].[2021-01-07]. https://yandex.ru/dev/mystem.
[11]	SEGALOVICH I. A fast morphological algorithm with unknown word guessing induced by a dictionary for a Web search engine [C]//International Conference on Machine Learning Models. DBLP, 2003. Las Vegas, Nevada: Springer, Cham, 2003: 273-280.
[12]	KOROBOV M. pymorphy2 [EB/OL]. [2021-01-07]. https://pypi.org/project/pymorphy2.
[13]	KHACHAY M Y, KONSTANTINOVA N, PANCHENKO A, et al. Morphological analyzer and generator for Russian and Ukrainian languages [C]//International Conference on Analysis of Images, Social Networks and Texts. Yekaterinburg, Russia:Springer, Cham, 2015: 320-332.
[14]	ЛУКАШЕВИЧ Н В, ГЕРАСИМОВА А А. Определе-ние устойчивых словосочетаний методом ассоциати-вного эксперимента[J]. Вестник Московского университета. Серия 9: Филология, 2018(1):23-42.
[15]	JACQUEMIN C. Recycling terms into a partial parser[C]// Fourth Conference on Applied Natural Language Processing. Stuttgart, Germany: Association for Computational Linguistics, 1994: 113-118.
[16]	CHURCH K W, HANKS P. Word association norms, mutual information, and lexicography[J]. Computational linguistics, 1990,16(1):22-29.
[17]	DICE L R. Measures of the amount of ecologic association between species[J]. Ecology, 1945,26(3):297-302. doi: 10.2307/1932409 URL
[18]	CHOUEKA Y. Looking for needles in a haystack or locating interesting collocational expressions in large textual databases [C]//Proceedings of the RIAO Conference on User-Oriented Content-Based Text and Image Handling, 1988, Cambridge, Mass, 1988: 609-623.
[19]	SILVA J F D, LOPES G P, TORRE Q D, et al. A local maxima method and a fair dispersion normalization for extracting multi-word units from corpora [C]//Sixth Meeting on Mathematics of Language. Orlando, USA, 1999: 369-381.
[20]	陈建超, 郑启伦, 李庆阳, 等. 基于词序列频率有向网的中文组合词提取算法[J]. 计算机应用研究, 2009,26(10):3746-3749.
[21]	龚双双, 陈钰枫, 徐金安, 等. 基于网络文本的汉语多词表达抽取方法[J]. 山东大学学报(理学版), 2018,53(9):40-48.
[22]	FRANTZI K, ANANIADOU S. Extracting nested collocations[C]// Proceedings of the 16th Conference on Computational Linguistics.Copenhagen,Denmark, 1996: 41-46.
[23]	唐亮, 李倩, 许洪波, 等. 基于多策略过滤的汉日多词短语抽取和对齐[J]. 山东大学学报(理学版), 2015,50(9):21-28.
[24]	马建红, 姬帅, 刘硕. 面向专利的主题短语提取[J]. 计算机工程与设计, 2019,40(5):1365-1369.
[25]	刘晨晖, 张德生, 胡钢. 基于Kert的中文主题关键短语提取算法[J]. 计算机应用, 2019,39(1):245-249.
[26]	RAHAMAN M M, AMIN M R. Language independent statistical approach for extracting keywords[C]//2017 4th International Conference on Advances in Electrical Engineering (ICAEE). Dhaka, Bangladesh: IEEE Press, 2017: 205-210.
[27]	RABBY G, AZAD S, MAHMUD M, et al. TeKET:a Tree-Based Unsupervised Keyphrase Extraction Technique[J]. Cognitive Computation, 2020,12(6):811-833. doi: 10.1007/s12559-019-09706-3 URL
[28]	DOBROV B V, LOUKACHEVITCH N V. Multiple evidence for term extraction in broad domains[C]// Recent Advances in Natural Language Processing, Hissar, Bulgaria, 2011: 710-715.
[29]	WESTLING A, BRYNIELSSON J, GUSTAVI T. Mining the web for sympathy: the pussy riot case[C]// 2014 IEEE Joint Intelligence and Security Informatics Conference. The Hague, Netherlands: IEEE, 2014: 123-128.
[30]	LAGUTINA K, LARIONOV V, PETRYAKOV V, et al. Sentiment classification of russian texts using automatically generated thesaurus[C]//Proceedings of the 23rd Conference of Open Innovations Association FRUCT. Bologna, Italy: IEEE Press, 2018: 13-16.
[31]	ХРАМЦОВ Н С. Проблематика оценивания алгорит-мов автоматического извлечения ключевых слов[J]. Новые информационные технологии в автоматизиро-ванных системах, 2019(22):199-203.
[32]	SCHMID H. TreeTagger-uni-muenchen.de[EB/OL]. [2021-03-31]. https://cental.uclouvain.be/treetagger.
[33]	BIRD S, KLEIN E, LOPER E. NLTK [EB/OL].[2020-04-13]. http://www.nltk.org.
[34]	ZIEMSKI M, JUNCZYS M, POULIQUEN B. The United Nations parallel corpus[C]// Language Resources and Evaluation in Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Portorož, Slovenia, 2016.
[35]	SHAVRINA T, SHAPOVALOVA O. Taiga Corpus[EB/OL]. [2020-06-14]. https://github.com/TatianaShavrina/taiga_site.

词性标记	词性说明	构词率	是否停用词性
A	形容词	92.37%	否
ADV	副词	92.98%	-
APRO	代词	<<1%	是
CONJ	连词	<<1%	是
INTJ	感叹词	<<1%	是
NUM	数词	<<1%	是
PART	助词	<<1%	是
PR	介词	81.71%	-
S	名词	92.23%	否
V	动词	67.51%	否

词性标记	词性说明	构词率	是否停用词性
A	形容词	92.37%	否
ADV	副词	92.98%	-
APRO	代词	<<1%	是
CONJ	连词	<<1%	是
INTJ	感叹词	<<1%	是
NUM	数词	<<1%	是
PART	助词	<<1%	是
PR	介词	81.71%	-
S	名词	92.23%	否
V	动词	67.51%	否

序号	频繁词串	中文翻译	出现频次
1	алгоритм добыча данные	数据挖掘算法	2
2	анализировать данные	分析数据	2
3	обучать набор	训练集	2
4	тестовый набор	测试集	2
5	закономерность	模式	2
6	с	和;的;来自	2

序号	频繁词串	中文翻译	出现频次
1	алгоритм добыча данные	数据挖掘算法	2
2	анализировать данные	分析数据	2
3	обучать набор	训练集	2
4	тестовый набор	测试集	2
5	закономерность	模式	2
6	с	和;的;来自	2

语料及处理方法	联合国平行语料		Taiga Corpus语料
语料及处理方法	本文方法	n-gram	本文方法	n-gram
自动提词数目	16091	37504	2095	3317
成词数目	14648	14648	1781	1781
准确率	91.03%	39.06%	85.01%	53.70%