Training-Free
Long-Video QA
Event-Level Memory
2026
MemoryCard
Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
A reusable video-memory framework that transforms sparse, fragmented frames into semantically coherent multimodal evidence for frozen vision-language models.
Self-read semantic units · High-density card rendering · Question-aware retrieval · Temporal reordering
Abstract
From sparse frame clues to high-density multimodal memories.
Answer-relevant evidence in long videos is often sparse, brief, and temporally dispersed. Existing efficiency methods still use isolated frames or frame-derived representations as the basic clue unit. MemoryCard instead self-reads a video and its aligned utterances, segments it into semantically coherent events, generates event-level video gists, selects representative visual moments, and renders them into unified cards for retrieval and question answering.
Motivation
Long-video QA has an evidence-unit problem.
Retrieving the right timestamp is not enough when each retrieved clue is semantically sparse and disconnected from the event that gives it meaning.
Frames are fragmented
An isolated frame records an instantaneous visual state, but usually omits event boundaries, spoken content, and surrounding temporal context.
Critical clues are transient
Objects, actions, text, and speech may appear only momentarily, while most frames in a long video are redundant or irrelevant.
Reasoning needs event context
VLMs benefit from compact evidence units that bind local details to a coherent topic, speech clue, and temporal span.
Frame-centric → Event-centric
MemoryCard changes what gets retrieved.
Instead of selecting disconnected frames, the retriever selects self-contained event memories. Each card combines representative visual moments with a topic-level gist, aligned utterances, and temporal metadata.
- Higher semantic density per visual input
- Explicit speech and temporal grounding
- Reusable across many questions from the same video
Method
A reusable video-memory bank in three stages.
The answering VLM remains frozen. Memory construction is question-agnostic; only retrieval and resolution allocation depend on the downstream question.
Self-read
Semantic unit construction
A self-read VLM segments the full video into content-aware sessions according to event, topic, and scene transitions, then aligns timestamped utterances.
Render
High-density Memory Cards
For each session, the model produces a topic-level gist and representative visual moments. These are rendered with speech clues and the temporal span into a unified image.
Retrieve & answer
Relevance-aware allocation
LongCLIP retrieves relevant cards. The top cards receive higher resolution, all selected cards are restored to temporal order, and the frozen VLM predicts the answer.
Question-agnostic constructionBuild once
Question-aware retrievalSelect per query
Mixed-resolution allocation4 high · 8 mid · 32 low
Temporal reorderingReason coherently
Experiments
Consistent improvements across backbones and benchmarks.
Select an answering backbone to compare the frozen base model with its MemoryCard-augmented variant. All values are accuracy (%).
Base model
+ MemoryCard
Same answering model, prompt format, decoding setup, answer extraction, and evaluation protocol.
| Backbone | Variant | Video-MME | MLVU | LongVideoBench |
|---|---|---|---|---|
| Qwen2-VL-7B | Base | 53.7 | 56.9 | 53.5 |
| Qwen2-VL-7B | + MemoryCard | 60.5 +6.8 | 65.7 +8.8 | 58.4 +4.9 |
| Qwen3-VL-8B | Base | 57.4 | 57.2 | 56.3 |
| Qwen3-VL-8B | + MemoryCard | 64.7 +7.3 | 66.5 +9.3 | 60.1 +3.8 |
| MiniCPM-V-4.5 | Base | 59.9 | 57.0 | 55.6 |
| MiniCPM-V-4.5 | + MemoryCard | 67.2 +7.3 | 69.4 +12.4 | 62.0 +6.4 |
Analysis
Why the representation works.
Ablations show that gains come from more than retrieval: semantic session construction, multimodal rendering, resolution allocation, and temporal organization all matter.
Rendering adds value beyond raw-frame retrieval
With Qwen3-VL, raw-frame retrieval reaches 60.8 on Video-MME, while the full MemoryCard representation reaches 64.7.
Semantic sessions outperform mechanical segmentation
Content-aware sessions better preserve coherent events than uniform-frame, fixed-length, or shot-based units.
Temporal order helps the answering VLM
After relevance retrieval, restoring the original chronology produces a more coherent reasoning sequence than relevance order or random order.
Mixed resolution balances detail and coverage
High-resolution cards preserve fine details, while low-resolution cards expand event coverage under a comparable visual-token budget.
Fine-Grained Task Analysis
MemoryCard improves all six task categories.
The gains are not confined to a single capability. Event-level multimodal evidence improves fine-grained perception and recognition while also strengthening OCR, information synopsis, counting, and reasoning.
Key takeaway Memory Cards preserve local visual details while supplying the broader event context needed to interpret them. Counting and reasoning remain comparatively challenging, indicating room for denser temporal coverage and stronger cross-card aggregation.
More evidence-use analysis
Resolution allocation, temporal ordering, and top-k selection determine how the constructed Memory Cards are consumed under a constrained visual budget.
Case Study
See how MemoryCard organizes long video evidence.
Switch between the question-agnostic self-read construction view and the question-conditioned retrieval view.
Retrieve → allocate → reorder
Question-conditioned visual evidence
The retriever ranks cards from the reusable memory bank, then organizes them into 4 high-resolution, 8 medium-resolution, and 32 low-resolution inputs. Answer-critical details receive more pixels without sacrificing broad temporal coverage.
Resources & Citation
Reproduce, extend, and cite MemoryCard.
The project page is fully static and the paper code is publicly available. Update the citation entry if a proceedings version becomes available.
BibTeX
@article{yang2026memorycard,
title = {MemoryCard: Topic-Aware Multi-Modal Clue
Compression for Long-Video Question Answering},
author = {Yang, Qing and Huang, Pengcheng and Li, Xinze
and Liu, Zhenghao and Yan, Yukun and Gu, Yu
and Yu, Ge and Li, Gang and Sun, Maosong},
journal = {arXiv preprint arXiv:2606.05917},
year = {2026}
}