Token-space bottleneck
Joint audio–visual streams quickly inflate context length, memory use, and Time to First Token for long recordings and repeated queries.
Under Submission
Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models
Compress long audio–visual recordings into reusable LoRA memory, then answer queries with zero multimodal tokens in the active context window.
University of Maryland, College Park
Omni2LoRA internalizes audio–visual recordings into reusable LoRA memory, replacing long token context with compact parameter memory for repeated querying.
0multimodal tokens at answer time
12×lower query TTFT, up to
30%rank budget used for main results
75%compression with strong robustness
Overview
Move multimodal memory out of the context window.
Omnimodal language models reason jointly over video and audio, but long recordings generate large synchronized token streams. Token pruning helps, yet aggressive compression can discard the temporal audio–visual anchors needed for coherent reasoning.
Omni2LoRA takes a different route: it processes a recording once, converts the recording into an instance-specific LoRA adapter, and reuses that compact parameter state for all later questions. The frozen omni model answers from text plus the adapter rather than repeatedly carrying audio and visual tokens through the active prompt.
Coherence bottleneck
Independent pruning can preserve salient unimodal features while removing the brief synchronized cues that only become meaningful across modalities.
Parameter-space memory
Omni2LoRA encodes the recording into a reusable adapter and learns which rank directions to keep under a fixed memory budget.
Method
Two stages: internalize, then compress.
Full-rank hypernetwork training captures the recording; coherence-aware rank allocation makes that memory compact without sacrificing the audio–visual dependencies that matter.
Stage 1: a frozen omni encoder produces intermediate representations consumed by a Perceiver hypernetwork to generate candidate LoRA slots. Stage 2: a scoring policy selects a fixed-budget subset and is optimized with coherence-aware GRPO using joint, visual-only, and audio-only reference rollouts.
Full-rank parametric internalization
A Perceiver hypernetwork maps synchronized audio–visual hidden states into layer-wise, recording-specific LoRA weights in a single forward pass while the omni backbone remains frozen.
Coherence-aware rank allocation
A trainable allocation policy selects a compact subset of candidate rank directions under a fixed budget instead of applying a uniform or magnitude-only rule.
Counterfactual advantage shaping
Joint audio–visual rewards are contrasted with visual-only and audio-only counterfactuals so the policy is rewarded for preserving information that depends on both modalities.
Zero-context repeated inference
The compact adapter is generated once and reused. Subsequent questions are answered with no audio or video tokens in the active context, turning recording encoding into an amortized setup cost.
Results
Consistent gains across three omni backbones.
Main comparison at a 30% retained-rank budget and 32 frames. Average combines DailyOmni, UGC-AVQA, OmniVideo, and WorldSense according to the paper’s reported metrics.
Qwen2.5-Omni-3B
47.3
average score
O-MARC45.8
Full tokens44.1
InteractiveOmni-4B
47.6
average score
O-MARC45.8
Full tokens46.0
Qwen2.5-Omni-7B
53.2
average score
O-MARC51.1
Full tokens47.2
Qwen2.5-Omni-7B
Detailed benchmark comparison
| Method | DailyOmni | UGC-AVQA | OmniVideo | WorldSense | Average |
|---|---|---|---|---|---|
| Full Tokens | 56.3 | 54.1 | 34.6 | 43.6 | 47.2 |
| OmniZip | 51.8 | 52.7 | 30.0 | 39.6 | 43.5 |
| OMAC | 53.6 | 53.2 | 30.9 | 42.4 | 45.0 |
| O-MARC | 60.4 | 64.6 | 35.2 | 44.0 | 51.1 |
| Omni2LoRA | 63.6 | 68.0 | 36.6 | 45.8 | 53.2 |
Compression robustness
UGC-AVQA accuracy
Omni2LoRA remains strongest as the memory constraint tightens, while token-pruning baselines degrade more sharply.
Long-recording scaling
Average score at 1,024 frames
Parametric internalization stays stable as recording length grows, while direct in-context inference collapses near context limits.
Efficiency
Compress once. Reuse many times.
Repeated questions expose the advantage of moving the recording into parameter memory: the initial internalization cost is paid once, while later queries avoid multimodal context processing.
Single-question TTFT · 7B
0.49s
Omni2LoRA vs. 6.03s for full-context inference and 3.45s for O-MARC.
After 5 queries · 7B
0.82s/q
Amortized query latency drops rapidly as the one-time adapter construction cost is spread across interactions.
Long-run plateau
≈0.43s/q
The reusable adapter removes repeated audio–visual encoding from the answer path.
Audio + Video
→Omni2LoRA Adapter
→Question 1 → Answer
Question 2 → Answer
Question N → Answer
Takeaways
A different compression axis for long-context omni models.
Parametric memoryreplaces repeated multimodal prompt context with recording-specific LoRA weights.
Coherence-aware compressionuses counterfactual audio-only and visual-only rollouts to protect cross-modal evidence.
Fixed-budget reusekeeps query-time multimodal token load at zero and amortizes setup over repeated interaction.
Citation
Omni2LoRA is currently under submission.
@misc{mathur2027omni2lora,
title = {Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models},
author = {Puneet Mathur and Manan Suri and Dinesh Manocha},
year = {2027},
note = {Under submission}
}