TY  - CHAP
U1  - Buchbeitrag
A1  - Koplenig, Alexander
A1  - Wolfer, Sascha
A1  - Müller-Spitzer, Carolin
ED  - Dębowski, Łukasz
ED  - Bentz, Christian
T1  - Studying Lexical Dynamics and Language Change via Generalized Entropies: The Problem of Sample Size
T2  - Information Theory and Language
N2  - Recently, it was demonstrated that generalized entropies of order α offer novel and important opportunities to quantify the similarity of symbol sequences where α is a free parameter. Varying this parameter makes it possible to magnify differences between different texts at specific scales of the corresponding word frequency spectrum. For the analysis of the statistical properties of natural languages, this is especially interesting, because textual data are characterized by Zipf’s law, i.e., there are very few word types that occur very often (e.g., function words expressing grammatical relationships) and many word types with a very low frequency (e.g., content words carrying most of the meaning of a sentence). Here, this approach is systematically and empirically studied by analyzing the lexical dynamics of the German weekly news magazine Der Spiegel (consisting of approximately 365,000 articles and 237,000,000 words that were published between 1947 and 2017). We show that, analogous to most other measures in quantitative linguistics, similarity measures based on generalized entropies depend heavily on the sample size (i.e., text length). We argue that this makes it difficult to quantify lexical dynamics and language change and show that standard sampling approaches do not solve this problem. We discuss the consequences of the results for the statistical analysis of languages.
KW  - Entropie
KW  - Sprachstatistik
KW  - Sprachwandel
KW  - Stichprobenumfang
KW  - Zipfsches Gesetz
KW  - Jensen-Shannon divergence
KW  - Zipf's law
KW  - generalized divergence
KW  - generalized entropy
KW  - sample size
KW  - text length
Y1  - 2020
UN  - https://nbn-resolving.org/urn:nbn:de:bsz:mh39-100033
SN  - 978-3-03936-027-7
SB  - 978-3-03936-027-7
U6  - https://doi.org/10.3390/e21050464
DO  - https://doi.org/10.3390/e21050464
SP  - 216
EP  - 233
PB  - MDPI
CY  - Basel
ER  -