Rethinking LLM-Guided Visual Question Answering

Abstract

Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual knowledge. While recent work leverages Large Language Models (LLMs) as knowledge sources, the question of how deeply such guidance should be integrated into a vision language architecture remains underexplored.

In this work, we generate structured reasoning guidance — visual_focus concepts, distractor elements, and external knowledge — using Llama 3.3 70B and systematically compare two integration strategies within a unified frozen ViT + FLAN-T5 framework on the OK-VQA benchmark: decision-level late fusion and attention-matrix injection.

Key Finding: Contrary to the prevailing assumption that deeper integration is superior, we find that shallow late fusion achieves the strongest results (39.4% exact match, +10.0 over the no-guidance baseline), outperforming deep attention modification (31.8%) under data-scarce fine-tuning. We further contribute an LLM-based augmentation pipeline that expands training data more than tenfold via question paraphrasing. Our findings indicate that integration depth must be matched to the data regime.

Some Questions Cannot Be Answered by Looking

Why OK-VQA needs external factual knowledge beyond pixel data

🍷

Example from OK-VQA

Question: "How old do you have to be in Canada to do this?"

Visible Content: Adults sitting around a table drinking wine from glasses.

Ground Truth Answer: 18 (Legal drinking age in Canada)

❌ Visual Baseline (No Guidance) "wine" / "adult"

Only sees salient pixels; cannot deduce the external legal statute.

✅ Our Late Fusion (LLM-Guided) "18"

Utilizes external knowledge context: "legal drinking age in Canada is 18".

Overall Framework & Architectures

End-to-end pipeline: Captioning, Structured Reasoning, and Downstream Integration

Figure 1: Overall LLM-guided VQA framework showing offline caption generation (BLIP), structured reasoning extraction (Llama 3.3 70B), guidance projection (CLIP), and downstream integration variants.

Decision-Level Late Fusion (Shallow)

Top Performer • 39.44% EM

The projected guidance vector $\mathbf{P}$ is concatenated with the global CLS token $\mathbf{z}_{\mathrm{cls}}$ of the frozen Vision Transformer:

\mathbf{f} = \mathbf{W}_f \, [\mathbf{z}_{\mathrm{cls}} \parallel \mathbf{P}] + \mathbf{b}_f, \quad \mathbf{h}_0 = \mathbf{W}_v \mathbf{f}

The fused feature vector $\mathbf{h}_0$ is prepended as a single prefix token to the FLAN-T5 input embeddings.

Vision Transformer stays completely frozen: Pretrained visual representations are perfectly preserved.
Minimal optimization burden: Only a single linear fusion layer and projection need to be trained.
No inference penalty: Guidance vector is precomputed offline.

Attention-Matrix Injection (Deep)

31.80% EM • +2.4% over baseline

The guidance vector $\mathbf{P}$ directly modifies the Query and Key projections inside the self-attention blocks of the last 6 ViT layers:

\mathbf{Q}' = \mathbf{Q} + \alpha g \mathbf{g}_q \mathbf{1}^{\top}, \quad \mathbf{K}' = \mathbf{K} + \alpha g \mathbf{g}_k \mathbf{1}^{\top}

where $\alpha$ is a learnable scale and $g = \sigma(\mathbf{w}_g^{\top} \hat{\mathbf{P}})$ is a per-sample gate. The modified ViT layers are unfrozen to allow joint adaptation.

Intuition: Aims to reshape how the image is inspected by steering attention patches.
Limitation under data scarcity: Unfreezing 6 layers with few samples destabilizes general visual features.
Global shift issue: The $\mathbf{1}^{\top}$ broadcast cannot provide localized spatial bounding.

Baseline VQA Architecture

Baseline • 29.44% EM

Standard cross-modal setup without external knowledge guidance:

\mathbf{h}_0 = \mathbf{W}_v \mathbf{z}_{\mathrm{cls}}, \quad \tilde{\mathbf{E}} = [\mathbf{h}_0 ; \mathbf{E}(\mathbf{x}_{\mathrm{ids}})]

Visual information is extracted solely from the image and passed as a prefix token to FLAN-T5.

Experimental Results

Evaluation on the augmented OK-VQA validation subset under identical fine-tuning

Main Results Comparison

Strategy	Depth	Exact Match (%)	Edit Sim. (%)	Token F1 (%)	BERTScore	$\Delta$ EM
Baseline (no guidance)	—	29.44	40.57	30.97	91.21	—
Late fusion (shallow) 🏆	shallow	39.44	48.97	41.34	92.14	+10.00
Attention injection	deep	31.80	42.66	33.33	91.00	+2.36

* All models evaluated with exact match, token F1, Levenshtein edit similarity, and BERTScore on the top-500 answer filtered split.

Multi-Metric Profile

Radar comparison across 4 evaluation metrics. Late fusion strictly encloses both baseline and deep attention injection.

Per-Question-Type Exact Match

Late fusion improves across all question categories. Attention injection helps on Where questions but drops on Who and Yes/No.

Error Distribution Analysis

Late fusion drops completely wrong answers from 63.1% to 53.2%, successfully converting errors into exact matches.

Qualitative Examples

The baseline names salient objects (e.g. "zebra", "wicker"), while guidance corrects predictions to the requested attribute or function ("stripe", "refrigerator").

Why Does Shallow Beat Deep?

Three technical factors explaining why decision-level fusion outperforms attention injection

🛡️

1. Preservation of Pretrained Representations

Late fusion leaves the frozen Vision Transformer computation untouched. Attention injection perturbs queries and keys in 6 of 12 layers and unfreezes them. In low-data regimes (~29k training samples), the model struggles to re-stabilize its visual representations, leading to degraded performance.

🎯

2. Granularity Mismatch

The guidance vector $\mathbf{P}$ is a single global direction (mean over CLIP concept embeddings). Broadcasting this single vector across all $N$ image patches ($\mathbf{1}^{\top}$) cannot express which region each concept refers to. Late fusion treats $\mathbf{P}$ as a global semantic summary at the decision level, perfectly matching its true granularity.

⚖️

3. Optimization Burden

Attention injection introduces per-layer projections, gates, and scaling factors that must be learned jointly with unfrozen ViT layers. With ~6 samples per class, the simpler hypothesis space of late fusion (a single linear fusion layer) avoids overfitting and wins cleanly.

Dataset & Quick Start

Published on Hugging Face Datasets with native Parquet support & in-browser viewer

Python (Hugging Face Datasets)

from datasets import load_dataset

# Load the OK-VQA LLM-augmented dataset (36,442 rows)
dataset = load_dataset("rishii100/okvqa-llm-augmented")

# Access sample fields
sample = dataset["train"][0]
print("Question:", sample["question"])
print("Visual Focus:", sample["visual_focus"])
print("Knowledge:", sample["knowledge"])
print("Answer:", sample["answer"])

Download Trained Checkpoints

from huggingface_hub import hf_hub_download

# Download best Late Fusion model checkpoint
model_path = hf_hub_download(
    repo_id="rishii100/knowledge-guided-vqa-models",
    filename="best_fusion_model.pth"
)
print("Downloaded checkpoint to:", model_path)

Features Schema: question_id (int64), image_id (int64), question (string), answer (string), caption (string), visual_focus (list[string]), distractors (list[string]), knowledge (list[string])

Browse Dataset on Hugging Face ↗ Browse Models on Hugging Face ↗

BibTeX Citation

BibTeX

@inproceedings{saha2026rethinking,
  title     = {Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?},
  author    = {Saha, Aneerban and Kumar, Vijay and D., Sameer Madhav and Srivastava, Priyam and Bansal, Dhruv},
  booktitle = {Proceedings of the International Conference on Applied Data and Computing Sciences (ADCIS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.17079}
}