Long short-term memory (LSTM) networks have recently shown remarkable performance in several tasks that are dealing with natural language generation, such as image captioning or poetry composition. Yet, only few works have analyzed text generated by LSTMs in order to quantitatively evaluate to which extent such artificial texts resemble those generated by humans. We compared the statistical structure of LSTM-generated language to that of written natural language, and to those produced by Markov models of various orders. In particular, we characterized the statistical structure of language by assessing word-frequency statistics, long-range correlations, and entropy measures. Our main finding is that while both LSTM- and Markov-generated texts can exhibit features similar to real ones in their word-frequency statistics and entropy measures, LSTM-texts are shown to reproduce long-range correlations at scales comparable to those found in natural language. Moreover, for LSTM networks, a temperature-like parameter controlling the generation process shows an optimal value-for which the produced texts are closest to real language-consistent across different statistical features investigated.

Lippi, M., Montemurro, M., Degli Esposti, M., Cristadoro, G. (2019). Natural Language Statistical Features of LSTM-Generated Texts. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 30(11), 3326-3337 [10.1109/TNNLS.2019.2890970].

Natural Language Statistical Features of LSTM-Generated Texts

Cristadoro G.
2019

Abstract

Long short-term memory (LSTM) networks have recently shown remarkable performance in several tasks that are dealing with natural language generation, such as image captioning or poetry composition. Yet, only few works have analyzed text generated by LSTMs in order to quantitatively evaluate to which extent such artificial texts resemble those generated by humans. We compared the statistical structure of LSTM-generated language to that of written natural language, and to those produced by Markov models of various orders. In particular, we characterized the statistical structure of language by assessing word-frequency statistics, long-range correlations, and entropy measures. Our main finding is that while both LSTM- and Markov-generated texts can exhibit features similar to real ones in their word-frequency statistics and entropy measures, LSTM-texts are shown to reproduce long-range correlations at scales comparable to those found in natural language. Moreover, for LSTM networks, a temperature-like parameter controlling the generation process shows an optimal value-for which the produced texts are closest to real language-consistent across different statistical features investigated.
Articolo in rivista - Articolo scientifico
Authorship attribution; entropy; long short-term memory networks; long-range correlations; natural language generation (NLG)
English
2019
30
11
3326
3337
8681285
open
Lippi, M., Montemurro, M., Degli Esposti, M., Cristadoro, G. (2019). Natural Language Statistical Features of LSTM-Generated Texts. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 30(11), 3326-3337 [10.1109/TNNLS.2019.2890970].
File in questo prodotto:
File Dimensione Formato  
10281-252845.pdf

accesso aperto

Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Dimensione 2.32 MB
Formato Adobe PDF
2.32 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/252845
Citazioni
  • Scopus 61
  • ???jsp.display-item.citation.isi??? 51
Social impact