publications

2025

1. Where is this coming from? Making groundedness count in the evaluation of Document VQA models

   Armineh Nourbakhsh, Siddharth Parekh, Pranav Shetty, and 3 more authors

   In Findings of the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: NLP in a Multicultural World, Apr 2025

   Abs

   Document VQA models have evolved at an impressive rate over the past few years, coming close to or matching human performance on some benchmarks. We argue that common evaluation metrics used by popular benchmarks do not account for the semantic and multimodal groundedness of a model’s outputs. As a result, hallucinations and major semantic errors are treated the same way as well-grounded outputs, and the evaluation scores do not reflect the reasoning capabilities of the model. In response, we propose a new evaluation methodology that accounts for the groundedness of predictions with regards to the semantic category of the output as well as the multimodal placement of the output within the input document. Our proposed methodology is parameterized in such a way that users can configure the score to their preferences. We validate our scoring methodology using human judgment and show its potential impact on existing popular leaderboards. Through extensive analyses, we demonstrate that our proposed method produces scores that are a better indicator of a model’s robustness, and tends to give higher rewards to better-calibrated answers.

2024

1. AliGATr: Graph-based layout generation for form understanding

   Armineh Nourbakhsh, Zhao Jin, Siddharth Parekh, and 2 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2024, Nov 2024

   Abs DOI

   Forms constitute a large portion of layout-rich documents that convey information through key-value pairs. Form understanding involves two main tasks, namely, the identification of keys and values (a.k.a Key Information Extraction or KIE) and the association of keys to corresponding values (a.k.a. Relation Extraction or RE). State of the art models for form understanding often rely on training paradigms that yield poorly calibrated output probabilities and low performance on RE. In this paper, we present AliGATr, a graph-based model that uses a generative objective to represent complex grid-like layouts that are often found in forms. Using a grid-based graph topology, our model learns to generate the layout of each page token by token in a data efficient manner. Despite using 30% fewer parameters than the smallest SotA, AliGATr performs on par with or better than SotA models on the KIE and RE tasks against four datasets. We also show that AliGATr’s output probabilities are better calibrated and do not exhibit the over-confident distributions of other SotA models.