Volltext-Downloads (blau) und Frontdoor-Views (grau)

Removing spam from web corpora through supervised learning using FastText

  • Unlike traditional text corpora collected from trustworthy sources, the content of web based corpora has to be filtered. This study briefly discusses the impact of web spam on corpus usability and emphasizes the importance of removing computer generated text from web corpora. The paper also presents a keyword comparison of an unfiltered corpus with the same collection of texts cleaned by a supervised classifier trained using FastText. The classifier was able to recognize 71% of web spam documents similar to the training set but lacked both precision and recall when applied to short texts from another data set.

Download full text files

Export metadata

Additional Services

Share in Twitter Search Google Scholar

Statistics

frontdoor_oas
Metadaten
Author:Vít Suchomel
URN:urn:nbn:de:bsz:mh39-62674
Parent Title (English):Proceedings of the Workshop on Challenges in the Management of Large Corpora and Big Data and Natural Language Processing (CMLC-5+BigNLP) 2017 including the papers from the Web-as-Corpus (WAC-XI) guest section. Birmingham, 24 July 2017
Publisher:Institut für Deutsche Sprache
Place of publication:Mannheim
Editor:Piotr Bański, Marc Kupietz, Harald Lüngen, Paul Rayson, Hanno Biber, Evelyn Breiteneder, Simon Clematide, John Mariani, Mark Stevenson, Theresa Sick
Document Type:Conference Proceeding
Language:English
Year of first Publication:2017
Date of Publication (online):2017/07/06
Publicationstate:Veröffentlichungsversion
Reviewstate:Peer-Review
Tag:Corpus linguistics; Corpus technology; Web corpora; Web spam
GND Keyword:Datenmanagement; Internet; Korpus <Linguistik>; Texttechnologie
Pagenumber:5
First Page:56
Last Page:60
Dewey Decimal Classification:400 Sprache
Open Access?:ja
Leibniz-Classification:Sprache, Linguistik
Linguistics-Classification:Korpuslinguistik
Conferences, Workshops:CMLC-5 + BigNLP / 5th Workshop on Challenges in the Management of Large Corpora and Big Data and Natural Language Processing
Licence (German):License LogoCreative Commons - Namensnennung-Nicht kommerziell-Keine Bearbeitung 3.0 Deutschland