Volltext-Downloads (blau) und Frontdoor-Views (grau)

Evaluating DBMS-based Access Strategies to Very Large Multi-layer Corpora

  • Linguistic query systems are special purpose IR applications. As text sizes, annotation layers, and metadata schemes of language corpora grow rapidly, performing complex searches becomes a highly computational expensive task. We evaluate several storage models and indexing variants in two multi-processor/multi-core environments, focusing on prototypical linguistic querying scenarios. Our aim is to reveal modeling and querying tendencies – rather than absolute benchmark results – when using a relational database management system (RDBMS) and MapReduce for natural language corpus retrieval. Based on these findings, we are going to improve our approach for the efficient exploitation of very large corpora, combining advantages of state-of-the-art database systems with decomposition/parallelization strategies. Our reference implementation uses the German DeReKo reference corpus with currently more than 4 billion word forms, various multi-layer linguistic annotations, and several types of text-specific metadata. The proposed strategy is language-independent and adaptable to large-scale multilingual corpora.

Export metadata

Additional Services

Share in Twitter Search Google Scholar


Author:Roman Schneider
Parent Title (English):Proceedings of the LREC-12 Workshop on Challenges in the Management of Large Corpora. Istanbul, Turkey, May 2012
Publisher:European Language Resources Association (ELRA)
Place of publication:Paris
Editor:Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Document Type:Conference Proceeding
Year of first Publication:2012
Date of Publication (online):2016/04/26
Tag:Concurrency; Database Management Systems; Linguistic Retrieval; Multi-layer Annotation; Very Large Corpora
First Page:35
Last Page:48
Dewey Decimal Classification:400 Sprache / 410 Linguistik
Leibniz-Classification:Sprache, Linguistik
Open Access?:Ja
Licence (German):Es gilt das UrhG