Title: Coreference Resolution on a Downstream Task: From Rule-based to State of the Art
Speaker: Haixia Chai (HITS)
Abstract
Coreference resolution is a key step in natural language understanding. Developments in coreference resolution are mainly focused on improving the performance on standard datasets annotated for coreference resolution. However, coreference resolution is an intermediate step for text understanding and it is not clear how these improvements translate into downstream task performance.
In this work, we perform a thorough investigation on the impact of coreference resolvers on a downstream task, i.e., answer selection with long answers. We first develop an evaluation framework for extrinsic evaluation of coreference resolvers on answer selection, and then annotate a subset of answers with coreference information for intrinsic evaluation. The results of our extrinsic evaluation show that (i) the impact of coreference resolution varies considerably on different answer selection models, and (ii) while there is a significant difference between the performance of the rule-based system vs. state-of-the-art neural model on coreference resolution datasets, we do not observe a considerable difference on their impact on downstream models. The results of intrinsic evaluation show that (i) resolving coreference relations on less-formal text genres is more difficult even for trained annotators, and (ii) the values of linguistic-agnostic coreference evaluation metrics do not correlate with the impact on downstream data.
