Refine
Year of publication
Document Type
- Conference Proceeding (16)
- Article (3)
- Part of a Book (3)
Language
- English (22) (remove)
Has Fulltext
- yes (22)
Keywords
- Korpus <Linguistik> (14)
- Deutsch (5)
- Automatische Sprachanalyse (4)
- Englisch (4)
- Sprachwandel (4)
- Deutsches Referenzkorpus (DeReKo) (3)
- Sprachvariante (3)
- Visualisierung (3)
- Automatische Sprachverarbeitung (2)
- Computerlinguistik (2)
Publicationstate
- Veröffentlichungsversion (12)
- Postprint (5)
- Zweitveröffentlichung (1)
Reviewstate
- Peer-Review (14)
- (Verlags)-Lektorat (2)
- Peer-Revied (1)
Publisher
- European Language Resources Association (ELRA) (4)
- Zenodo (2)
- Association for Computational Linguistics (1)
- Deutsche Gesellschaft für Sprachwissenschaft (1)
- European Language Resources Association (1)
- European language resources association (ELRA) (1)
- Frontiers Media S.A. (1)
- Jagiellonian University; Pedagogical University (1)
- Lancaster University (1)
- Leibniz-Institut für Deutsche Sprache (1)
We present an approach for automatic detection and correction of OCR-induced misspellings in historical texts. The main objective is the post-correction of the digitized Royal Society Corpus, a set of historical documents from 1665 to 1869. Due to the aged material the OCR procedure has made mistakes, thus leading to files corrupted by thousands of misspellings. This motivates a post processing step. The current correction technique is a pattern-based approach which due to its lack of generalization suffers from bad recall.
To generalize from the patterns we propose to use the noisy channel model. From the pattern based substitutions we train a corpus specific error model complemented with a language model. With an F1-Score of 0.61 the presented technique significantly outperforms the pattern based approach which has an F1-score of 0.28. Due to its more accurate error model it also outperforms other implementations of the noisy channel model.
We analyze the linguistic evolution of selected scientific disciplines over a 30-year time span (1970s to 2000s). Our focus is on four highly specialized disciplines at the boundaries of computer science that emerged during that time: computational linguistics, bioinformatics, digital construction, and microelectronics. Our analysis is driven by the question whether these disciplines develop a distinctive language use—both individually and collectively—over the given time period. The data set is the English Scientific Text Corpus (scitex), which includes texts from the 1970s/1980s and early 2000s. Our theoretical basis is register theory. In terms of methods, we combine corpus-based methods of feature extraction (various aggregated features [part-of-speech based], n-grams, lexico-grammatical patterns) and automatic text classification. The results of our research are directly relevant to the study of linguistic variation and languages for specific purposes (LSP) and have implications for various natural language processing (NLP) tasks, for example, authorship attribution, text mining, or training NLP tools.
Language resources are often compiled for the purpose of variational analysis, such as studying differences between genres, registers, and disciplines, regional and diachronic variation, influence of gender, cultural context, etc. Often the sheer number of potentially interesting contrastive pairs can get overwhelming due to the combinatorial explosion of possible combinations. In this paper, we present an approach that combines well understood techniques for visualization heatmaps and word clouds with intuitive paradigms for exploration drill down and side by side comparison to facilitate the analysis of language variation in such highly combinatorial situations. Heatmaps assist in analyzing the overall pattern of variation in a corpus, and word clouds allow for inspecting variation at the level of words.