Korpuslinguistik
Refine
Year of publication
Document Type
- Conference Proceeding (85)
- Article (62)
- Part of a Book (42)
- Book (9)
- Part of Periodical (2)
- Review (2)
Keywords
- Korpus <Linguistik> (175)
- Deutsch (51)
- Annotation (26)
- Forschungsdaten (26)
- Gesprochene Sprache (25)
- corpus linguistics (22)
- Datenmanagement (20)
- Computerlinguistik (15)
- Corpus linguistics (15)
- Corpus technology (12)
Publicationstate
- Veröffentlichungsversion (154)
- Zweitveröffentlichung (42)
- Postprint (10)
Reviewstate
- Peer-Review (202) (remove)
Publisher
- European Language Resources Association (20)
- Institut für Deutsche Sprache (20)
- de Gruyter (12)
- Leibniz-Institut für Deutsche Sprache (11)
- Linköping University Electronic Press (10)
- CLARIN (8)
- Universitäts- und Landesbibliothek Darmstadt (7)
- European Language Resources Association (ELRA) (6)
- IDS-Verlag (6)
- Peter Lang (6)
Common Crawl is a considerably large, heterogeneous multilingual corpus comprised of crawled documents from the internet, surpassing 20TB of data and distributed as a set of more than 50 thousand plain text files where each contains many documents written in a wide variety of languages. Even though each document has a metadata block associated to it, this data lacks any information about the language in which each document is written, making it extremely difficult to use Common Crawl for monolingual applications. We propose a general, highly parallel, multithreaded pipeline to clean and classify Common Crawl by language; we specifically design it so that it runs efficiently on medium to low resource infrastructures where I/O speeds are the main constraint. We develop the pipeline so that it can be easily reapplied to any kind of heterogeneous corpus and so that it can be parameterised to a wide range of infrastructures. We also distribute a 6.3TB version of Common Crawl, filtered, classified by language, shuffled at line level in order to avoid copyright issues, and ready to be used for NLP applications.
Nearly all of the very large corpora of English are “static”, which allows a wide range of one-time, pre-processed data, such as collocates. The challenge comes with large “dynamic” corpora, which are updated regularly, and where preprocessing is much more difficult. This paper provides an overview of the NOW corpus (News on the Web), which is currently 8.2 billion words in size, and which grows by about 170 million words each month. We discuss the architecture of NOW, and provide many examples that show how data from NOW can (uniquely) be extracted to look at a wide range of ongoing changes in English.
As the Web ought to be considered as a series of sources rather than as a source in itself, a problem facing corpus construction resides in meta-information and categorization. In addition, we need focused data to shed light on particular subfields of the digital public sphere. Blogs are relevant to that end, especially if the resulting web texts can be extracted along with metadata and made available in coherent and clearly describable collections.
This paper reports on the latest developments of the European Reference Corpus EuReCo and the German Reference Corpus in relation to three of the most important CMLC topics: interoperability, collaboration on corpus infrastructure building, and legal issues. Concerning interoperability, we present new ways to access DeReKo via KorAP on the API and on the plugin level. In addition we report about advancements in the EuReCo- and ICC-initiatives with the provision of comparable corpora, and about recent problems with license acquisitions and our solution approaches using an indemnification clause and model licenses that include scientific exploitation.
Contents:
1. Johannes Graën, Tannon Kew, Anastassia Shaitarova and Martin Volk, "Modelling Large Parallel Corpora", S. 1-8
2. Pedro Javier Ortiz Suárez, Benoît Sagot and Laurent Romary, "Asynchronous Pipelines for Processing Huge Corpora on Medium to Low Resource Infrastructures", S. 9-16
3. Vladimír Benko, "Deduplication in Large Web Corpora", S. 17-22
4. Mark Davies, "The best of both worlds: Multi-billion word “dynamic” corpora", S. 23-28
5. Adrien Barbaresi, "On the need for domain-focused web corpora", S. 29-32
6. Marc Kupietz, Eliza Margaretha, Nils Diewald, Harald Lüngen and Peter Fankhauser, "What's New in EuReCo? Interoperability, Comparable Corpora, Licensing", S. 33-39
Distributional models of word use constitute an indispensable tool in corpus based lexicological research for discovering paradigmatic relations and syntagmatic patterns (Belica et al. 2010). Recently, word embeddings (Mikolov et al. 2013) have revived the field by allowing to construct and analyze distributional models on very large corpora. This is accomplished by reducing the very high dimensionality of word cooccurrence contexts, the size of the vocabulary, to few dimensions, such as 100-200. However, word use and meaning can vary widely along dimensions such as domain, register, and time, and word embeddings tend to represent only the most prevalent meaning. In this paper we thus construct domain specific word embeddings to allow for systematically analyzing variations in word use. Moreover, we also demonstrate how to reconstruct domain specific co-occurrence contexts from the dense word embeddings.
Die korpusbasierte Lexikografie ist ein interessanter und vielfältiger wissenschaftlicher Anwendungsbereich, der auch im muttersprachlichen Deutschunterricht und im Deutsch-als-Fremdsprache-Unterricht eine größere Rolle einnehmen sollte. In unserem Beitrag stellen wir deshalb geeignete Korpora und Korpusanalysewerkzeuge vor, mit deren Hilfe Nutzerinnen und Nutzer einzelne Angabebereiche in einem Wörterbuch nicht nur nachvollziehen, sondern auch eigenständig erarbeiten können. Neben vorhandenen Ansätzen geschieht dies am Beispiel des Denktionarys, eines wikibasierten Wörterbuches, für das Schülerinnen und Schüler im Rahmen des Projekts Schüler machen Wörterbücher – Wörterbücher machen Schule im muttersprachlichen Deutschunterricht selbst korpusbasierte Artikel verfassten.
Ein sehr mächtiges Instrument für die Untersuchung von Wörtern und Verwandtschaftsbeziehungen zwischen ihnen ist die Analyse typischer Verwendungskontexte - unabhängig davon, ob die Evidenzen auf Bedeutungskonstitution, ihre Veränderung oder Verwechslung hinweisen, drei Aspekte, die alle bei der Charakterisierung von Paronymie eine Rolle spielen. Auch wenn für die Ermittlung typischer Verwendungsmuster ausgereifte Methoden zur Verfügung stehen, so sollte beim Vergleich der Analysen doch beachtet werden, dass sie diversen Einflussgrößen unterliegen. Neben der Datengrundlage und der Definition und Handhabung des relevanten Kontextes wird im Folgenden besonders darauf eingegangen, welche Rolle verschiedene Teilmengen eines Flexionsparadigmas spielen können, wenn ein Lemma als dessen Gesamtmenge als sprachliche Bezugseinheit einer Untersuchung gewählt wurde. Veranschaulicht wird die Gedankenführung an der beispielhaften Betrachtung von Paronymkandidaten.
This contribution presents a quantitative approach to speech, thought and writing representation (ST&WR) and steps towards its automatic detection. Automatic detection is necessary for studying ST&WR in a large number of texts and thus identifying developments in form and usage over time and in different types of texts. The contribution summarizes results of a pilot study: First, it describes the manual annotation of a corpus of short narrative texts in relation to linguistic descriptions of ST&WR. Then, two different techniques of automatic detection – a rule-based and a machine learning approach – are described and compared. Evaluation of the results shows success with automatic detection, especially for direct and indirect ST&WR.