Weighted error — new metrics for estimating quality of answer validation in the problem of question-answering retrieval

The answer validation subproblem is considered in a problem of question answering retrieval. Traditional quality metrics at the TAC-RTE and CLEF-AVE seminars are accuracy and F-measure. From results of participation in ROMIP-2010 seminar, it is noted that a number of false answer-hypotheses that must be declined by the answer validation module frequently exceeds substantially the true answer number. A novel metrics—weighted error is proposed which penalizes the system for the first-kind errors (false positive errors, when an incorrect answer is shown to the user) more frequently than for the second-kind errors (false negative errors, when a correct answer is rejected and not shown to the user). Unlike the F-measure, it also rewards the system for the properly rejected (true negative) answer.

References

[1] Соловьев А.А. Алгоритмы валидации ответов в задаче вопросно-ответного поиска // Вестник Воронежского гос. ун-та. Сер.: Системный анализ и информационные технологии. – 2011. – № 2. – C. 181–188.

[2] Voorhees E. The TREC-8 question answering track report // In Proc. of the Eighth Text REtrieval Conference (TREC 8). – 1999. – P. 77–82.

[3] Voorhees E.M. Overview of the TREC 2002 question answering track // In Proc. of the Eleventh Text Retrieval Conference (TREC 2002). – P. 57–67.

[4] Voorhees E.M. Overview of the TREC 2004 question answering track // In Proc. of The Thirteenth Text Retrieval Conference (TREC 2004).

[5] Peсas А., Hovy E., Forner P., Rodrigo A., Sutcliffe R., Forascu C. and Sporleder C. Overview of QA4MRE at CLEF 2011: Question answering for machine reading evaluation // Working Notes for the CLEF 2011 Workshop. – 2011.

[6] Соловьев А.А. Кто виноват и где собака зарыта? Метод валидации ответов на основе неточного сравнения семантических графов в вопросно-ответной системе // Российский семинар по оценке методов информационного поиска: Тр. РОМИП 2010. (Казань, 15 октября 2010 г.).

[7] Wang Neumann. Using recognizing textual entailment as a core engine for answer validation // Working Notes for the CLEF 2008 Workshop. – 2008.

[8] Marsi E., Krahmer E., Bosma W.E., Theune M. Normalized alignment of dependency trees for detecting textual entailment // Second PASCAL Recognising Textual Entailment Challenge. – 10–12 April 2006. – Venice, Italy.

[9] Punyakanok V., Roth D. and Yih W. Natural language interface via dependency tree mapping: An application to question answering // AI and Math. – January, 2004.

[10] Schlaefer N. A semantic approach to question answering. Saarbrucken 2007.