Leibniz MMS Days 2024 - Invited Talk

Demberg, Vera (Saarland University)

Pragmatic processing in large language models

In linguistics, pragmatics refers to meaning that is intended but not spelled out literally, i.e. it concerns what we humans understand beyond the semantic meaning of a sentence or text. Prominent examples of these include for instance scalar implicatures, such as the utterance "Some students failed", which is in its literal meaning compatible with a situation where all students failed, but which is usually interpreted as "some but not all" students having failed. Traditionally, these phenomena are modelled using game-theoretic models of human interaction (such as the rational speech act (RSA) model, Frank and Goodman, 2012). In my talk, I will provide an overview of how well recent large language models like ChatGPT are performing on pragmatic tasks - largely showing that they fail in many respects, and will then proceed to analyse what may be lacking from current LLMs.