title: Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy
abstract: Meta-evaluation of automatic evaluation metrics-assessing evaluation metrics themselves-is crucial for accurately benchmarking natural language processing systems and has implications for scientific inquiry, production model development, and policy enforcement.While existing approaches to metric meta-evaluation focus on general statements about the absolute and relative quality of metrics across arbitrary system outputs, in practice, metrics are applied in highly contextual settings, often measu