Transformer-Based Composite Language Models for Text Evaluation and Classification

Објеката

Тип
Рад у часопису
Верзија рада
објављена верзија
Језик
енглески
Креатор
Mihailo Škorić, Miloš Utvić, Ranka Stanković
Извор
Mathematics
Издавач
MDPI AG
Датум издавања
2023
Сажетак
Parallel natural language processing systems were previously successfully tested on the tasks of part-of-speech tagging and authorship attribution through mini-language modeling, for which they achieved significantly better results than independent methods in the cases of seven European languages. The aim of this paper is to present the advantages of using composite language models in the processing and evaluation of texts written in arbitrary highly inflective and morphology-rich natural language, particularly Serbian. A perplexity-based dataset, the main asset for the methodology assessment, was created using a series of generative pre-trained transformers trained on different representations of the Serbian language corpus and a set of sentences classified into three groups (expert translations, corrupted translations, and machine translations). The paper describes a comparative analysis of calculated perplexities in order to measure the classification capability of different models on two binary classification tasks. In the course of the experiment, we tested three standalone language models (baseline) and two composite language models (which are based on perplexities outputted by all three standalone models). The presented results single out a complex stacked classifier using a multitude of features extracted from perplexity vectors as the optimal architecture of composite language models for both tasks.
том
11
Број
22
doi
10.3390/math11224660
issn
2227-7390
Subject
General Mathematics, Engineering (miscellaneous), Computer Science (miscellaneous)
Шира категорија рада
M20
Ужа категорија рада
М21а
Права
Отворени приступ
Лиценца
All rights reserved
Формат
.pdf

Mihailo Škorić, Miloš Utvić, Ranka Stanković. "Transformer-Based Composite Language Models for Text Evaluation and Classification" in Mathematics, MDPI AG (2023). https://doi.org/10.3390/math11224660

This item was submitted on 17. новембар 2023. by [anonymous user] using the form “Рад у часопису” on the site “Радови”: http://drug.rgf.bg.ac.rs/s/repo

Click here to view the collected data.