Using LLMs for experimental stimulus pretests in linguistics. Evidence from semantic associations between words and social gender
- Whether large language models (LLMs) can validly complement or substitute human participants in experimental research remains an open question. Focusing on language cognition, we assess the suitability of GPT-4o and LLaMA 3.1 models (70B Instruct and 8B Instruct) for performing a semantic-association task in German. LLMs labeled noun phrases by social-gender association and rated association strength, mirroring a human participant task. Overall, LLM ratings aligned with human data, but item-level analyses revealed systematic deviations in response patterns.
Download full text files
Additional Services
Statistics
| Author: | Christian LangORCiDGND, Franziska KretzschmarORCiDGND, Sandra HansenORCiDGND |
|---|---|
| URN: | urn:nbn:de:bsz:mh39-134338 |
| URL: | https://aclanthology.org/2025.konvens-1.28/ |
| Parent Title (English): | KONVENS : 21th Conference on Natural Language Processing (KONVENS 2025) ; Proceedings of the Conference. Volume 1: Long and Short Papers |
| Publisher: | HsH Applied Academics |
| Place of publication: | Hannover |
| Editor: | Christian WartenaORCiDGND, Ulrich HeidORCiDGND |
| Document Type: | Part of a Book |
| Language: | English |
| Year of first Publication: | 2025 |
| Date of Publication (online): | 2025/09/11 |
| Publishing Institution: | Leibniz-Institut für Deutsche Sprache (IDS) |
| Publicationstate: | Veröffentlichungsversion |
| Reviewstate: | Peer-Review |
| Tag: | GPT-4o; LLaMA 3.1; noun phrases; semantic-association task; social gender |
| GND Keyword: | Deutsch; Semantik; large language models |
| First Page: | 326 |
| Last Page: | 332 |
| DDC classes: | 400 Sprache / 400 Sprache, Linguistik |
| Open Access?: | ja |
| BDSL-Classification: | Grammatik |
| Linguistics-Classification: | Soziolinguistik |
| Program areas: | Grammatik |
| Licence (German): |  Creative Commons - CC BY - Namensnennung 4.0 International |