DocXEval
The first systematic evaluation framework for explainability (XAI) in multimodal document understanding. It comprises a benchmark dataset, evaluation metrics, and a meta-evaluation method for assessing the faithfulness and explainability of explanation methods.
Timeline (last 90 days)
DocXEval is the first systematic evaluation framework for explainability (XAI) in multimodal document understanding.
DocXEval systematically evaluates 13 XAI methods, 3 types of document understanding tasks, 8 datasets, 4 models and 17 evaluation metrics.
The evaluation results show that only a few methods maintain consistent performance across all evaluation criteria, among which Occlusion performs best overall on the two dimensions of faithfulness and interpretability.
DocXEval's meta-evaluation found moderate to strong agreement among several faithfulness metrics and complexity metrics, while also identifying some metrics that produce unreliable results.
DocXEval's code is publicly available on GitHub at https://github.com/saifullah3396/docxeval/.
For tasks such as semantic entity recognition, DocXEval introduces efficient multi-target adapted versions of explanation methods and metrics.
All Facts (6)
DocXEval is the first systematic evaluation framework for explainability (XAI) in multimodal document understanding.
50%UnverifiedDocXEval systematically evaluates 13 XAI methods, 3 types of document understanding tasks, 8 datasets, 4 models and 17 evaluation metrics.
50%UnverifiedThe evaluation results show that only a few methods maintain consistent performance across all evaluation criteria, among which Occlusion performs best overall on the two dimensions of faithfulness and interpretability.
50%UnverifiedDocXEval's meta-evaluation found moderate to strong agreement among several faithfulness metrics and complexity metrics, while also identifying some metrics that produce unreliable results.
50%UnverifiedFor tasks such as semantic entity recognition, DocXEval introduces efficient multi-target adapted versions of explanation methods and metrics.
50%UnverifiedDocXEval's code is publicly available on GitHub at https://github.com/saifullah3396/docxeval/.
50%