State Space Model based Temporal Adaptive Token Compression for Efficient Multimodal Large Language Model

1. Large Multimodal Model and Video Temporal Understanding

LMMs allow comprehensive video understanding by combining visual, textual, and auditory features [11]. Earlier approaches introduced projection layers and Q-Former to map the output from visual encoders, incorporating additional parameters such as gated cross-attention and adapter layers, to the baseline models to process multimodal inputs [12, 13]. However, token limitations of the LLMs force the adoption of sparse sampling, which inherently overlook fine-grained temporal information and immediate visual details [2].

Several studies have proposed adaptive frame processing techniques to overcome these limitations. Video-ChatGPT [3] introduced the frame compression method that utilizes spatio-temporal pooling to aggregate information efficiently. LLaVA-MR [4] proposed an approach to capture detailed temporal representation through time encoding and select informative frames using a variance-based dynamic compression. While these approaches have partially alleviated the problem of limited token input, a lack of integrated processing remains that leverages the interdependence between frame selection and the temporal context of the video.

2. Token Compression for Video Representation

An efficient token reduction approach is essential in video processing due to the quadratic computational complexity of transformers. ToMe improves throughput by a factor of two without retraining by combining duplicate tokens using a bipartite matching algorithm based on attention key similarity. TokenLearner proposes an adaptive tokenization method that compresses hundreds of input tokens into 8–16 semantically salient representations using an attention mechanism.

In this study, the STAC module dynamically evaluates the saliency of visual tokens generated by the Q-Former of the LMM before projecting them to the backbone, by selectively compressing the core information. Through the long-term dependency modeling of Mamba, the STAC compresses visual tokens based on global temporal context, enabling efficient computation without sacrificing semantically critical visual details.

1. Model Architecture

We introduce a lightweight video temporal understanding framework, as shown in Fig. 2. The extracted video representation from Q-Former is processed along the temporal axis using a compression module. The STAC effectively reduces the sequence length of vision embeddings by extracting only the salient tokens. This approach reduces the dimensionality of the Q-Former output and significantly shortens the LLM [13] input sequence length, thereby mitigating the input bottleneck while improving computational efficiency and reducing inference time.

Fig. 2.

We use a 4-bit quantized BLIP-2 [9] that encodes video and text queries. The videos are embedded via a frozen vision encoder and processed through Q-Former to produce video representations. STAC reduces the 32 dimensions of the Q-Former outputs to 16 temporally salient tokens. After projection and concatenation with text embeddings, the model processes these through a LoRA-adapted LLM [13] for end-to-end fine-tuning.

First, the vision encoder extracts uniformly sampled frames, and then reconstructed visual pixel values are processed to generate a frame-by-frame embedding. It processes [B×N, C, H, W], where the dimensions represent batch size, number of frames, channels, height, and width. The output frame embeddings of the vision encoder contain the patch tokens for each frame and finally consist the form F ∈ R^B×N×P×D, where P is the number of patch tokens per frame, and D is the embedding dimension. They are flattened to a single sequence and fed into Q-Former. Q-Former utilizes learnable query tokens to extract high-level features. Our proposed Mamba-based token compression processes the video representation of the Q-Former, which selects the most salient Top-K tokens along the temporal axis of the form [B×N×K×D].

The text encoder processes the target query embedding form of T ∈ R^B×L×D, where L denotes the sequence length. The visual tokens selected by the STAC are mapped to the same dimensional space as the text embeddings through a projection layer, after which the two modalities construct an integrated multimodal representation. Finally, the LLM generates the temporal localization output through the integrated representation in the following format: “It starts at {start time: X} and ends at {end time: Y} seconds.”

2. State space model-based Temporal Adaptive Compression

The STAC selects temporally salient video representation tokens from Q-Former outputs for efficient fine-tuning while preserving the knowledge of pre-trained LMM models. Projecting the large number of Q-Former output tokens into the LLM input causes a computational bottleneck. Our approach addresses it by selectively retaining only the most temporally salient tokens. STAC prunes less informative tokens by leveraging attention and inter-frame relational analysis and decreases the amount of computation after projection without compromising the saliency of the video tokens.

Fig. 3 presents the detailed architecture of the STAC module. The input of STAC is [B×N×Q×D], where Q indicates the number of Q-Former output tokens, and D corresponds to the feature dimensionality of each token. The original representation is reshaped into a sequential format [B×Q, N, D] through a view operation. We add a positional encoding, normalize the value, and input into the Mamba block. Mamba employs an input-conditioned transition matrix to encode sequential patterns across N time steps into a compressed state representation, preserving salient information while eliminating redundant components [8].

Fig. 3.

The Q-Former outputs receive temporal position encoding before Mamba processing to model sequence dynamics. The video tokens undergo saliency assessment through an attention layer with learnable queries. Based on the computed saliency rankings, the module retains only the top-K most relevant tokens for subsequent stages.

After generating the key tensor [B×Q, N, D] via projection, we calculate the attention weights [B×Q, H, D] using four learnable queries [H, D]. Applying softmax over the temporal axis of N, each attention head generates a normalized weight distribution that highlights salient temporal localization in the sequence. Each head produces a summary vector that captures temporal information, resulting in head-specific features [B×Q, H, D]. These are flattened [B×Q, H×D] MLP to output a scalar score. Within each batch, we apply Top-K filtering to isolate the highest-scoring query localization. These positions are arranged in chronological order to ensure sequence integrity and facilitate feature extraction from the original embedding space. This process results in an output embedding with dimensions [B, N, K, D]

3. Training and Inference

We employ the STAC to extract the key visual tokens that compose the video representation, reducing computation by quantizing the pre-trained model to 4-bit and incorporating low-rank matrices into some LLM layers with LoRA for efficient training. LoRA preserves the weight W₀ and only trains the low-rank update weights ΔW = BA. LoRA achieves video-to-text alignment by fine-tuning visual and textual tokens together through low-rank matrix decomposition, which preserves the pretrained weights while generating fine-tuned weight W₀+ΔW applied during inference for efficient processing [10].

We define the following loss function for fine-tuning.

where we use the model maximum likelihood loss minimization over input x and target y pairs of length l, then use greedy decoding during inference to generate temporal descriptions from which predicted start and end times are parsed and compared against ground truth for performance evaluation.

1. Experimental Setup

Datasets - we assess the performance of our module with Charades-STA [14], which serves as the primary evaluation for temporal localization tasks. It comprises 16,128 annotated samples containing segments with an average length of 8.1 seconds and text queries consisting of 7.22 words on average, extracted from videos with a mean duration of 30.6 seconds. This distribution includes 12,408 training samples and 3,820 test samples.

Metric - we employ conventional assessment for video temporal grounding of Recall@K and Intersection over Union (IoU) metrics. The mean IoU(mIoU) computes the average overlap ratio between predicted and ground truth across all samples. Recall@K quantifies the fraction of Top-K predictions that surpass a specified IoU threshold. We report R1@0.5 and R1@0.7 as accuracy measures for IoU thresholds of 0.5 and 0.7, respectively.

Implementation Details - we fine-tune 1.22% of the total parameters. We employ the AdamW optimization combined with WarmupCosineLR scheduling, configuring the learning rate at 1e-5 and weight decay at 0.01. The learning rate schedule incorporates a linear warmup phase that gradually increases from zero to peak value during the initial 10% of training iterations, followed by the cosine decay over the remaining 90%. We conduct experiments on a single A6000-48GiB GPU, processing 16 frames per video sequence and implementing early stopping across 20 training epochs. The training process identified epoch 11 as the optimal stopping point due to overfitting in later epochs, with the entire training approximately 19 hours. We hold the same experimental conditions in the tested methods for fair comparisons.

2. Performance Evaluation

2.1 Performance Comparison with Existing Approaches

In this study, we evaluate the performance and efficiency of our proposed STAC method within an LMM [9] architecture. We compare it against conventional token compression approaches: ToMe [5] and TokenLearner [6]. While ToMe merges similar vision transformer tokens and TokenLearner employs learnable queries for compression, STAC takes a different approach, which selects the Top-K most temporally salient tokens from the final concatenated video representation before feeding them into the LLM [13].

The STAC method demonstrated superior performance. It achieved an mIoU of 30.2, R1@0.5 of 34.3, and R1@0.7 of 17.1. In contrast, conventional token compression methods vary in degrees of performance degradation. TokenLearner exhibited the most significant performance degradation, with its mIoU of 22.1%, R1@0.5 of 25.2, and R1@0.7 of 13.2%. ToMe, while outperforming TokenLearner, still showed a degraded performance with mIoU with 28.2, R1@0.5 of 31.7, and R1@0.7 of 16.2.

Table 1.

Performance Comparison with Existing Approaches

We evaluated the computational efficiency of the model using FLOPs, MACs, and number of parameters, as summarized in Table 2. FLOPs and MACs directly represent the computational complexity of the model and were measured using the Calflops package. When TokenLearner was applied, TFLOPs increased slightly to 15.57 and TMACs to 7.78, indicating that the module imposes computational overhead. ToMe showed the highest values with 16.59 TFLOPs and 8.30 TMACs, suggesting that the token merging process caused higher-than-expected computational costs. On the other hand, the STAC achieved the lowest computational complexity with 12.51 TFLOPs and 6.25 TMACs. When comparing the number of model parameters, TokenLearner had the highest at 151.65 million, followed by STAC with 149.48 million. ToMe maintained the same 143.67 million parameters. While parameters of STAC are slightly higher than ToMe, it is lower than TokenLearner. It suggests an appropriate level of overhead for balancing efficiency and performance improvements.

Table 2.

Computational Complexity and Efficiency Comparison

In addition, we report inference time. TokenLearner used 0.37 seconds, and ToMe used 0.35 seconds, showing slight improvements. Due to a reduction in attention computations, it results from a decrease in the number of tokens. STAC recorded the fastest inference time of 0.30 seconds all methods. It demonstrates that enhancing computational efficiency leads to improved inference speed and reduced resource consumption.

2.2 Performance Analysis

To determine the optimal number of temporal tokens for our STAC framework, we conduct an experiment examining the impact of Top-K selection on model performance. We evaluate K = {4, 8, 16}, representing different degrees of temporal compression.

The experimental results in Table 3 show the results across different K values. K=4 achieves an mIoU of 31.09, while K=8 shows a significant performance drop to 27.58. K=16 recovers to achieve the highest performance at 31.28. At K=4, the model demonstrates stable performance through complete compression, selecting only the most essential temporal information. This aggressive compression eliminates redundancy while maintaining core temporal patterns. The intermediate value K=8 yields the poorest performance across all metrics, representing a suboptimal region where the model fails to make an appropriate balance between temporal compression efficiency and comprehensive temporal representation. K=8 provides insufficient tokens for capturing the full temporal diversity needed for complex moment localization, while simultaneously offering too many tokens to achieve the focused compression benefits seen at K=4. At K=16, the model achieves the best overall performance, particularly excelling in precise matching tasks (R1@0.5: 34.21). This configuration provides sufficient temporal diversity and coverage, enabling comprehensive preservation of temporal relationships across different time scales. These findings indicate that effective temporal compression operates optimally at two distinct regimes: either complete compression (K=4) for maximum efficiency, or sufficient expansion (K=16) to guarantee comprehensive temporal coverage. Based on these results, we adopt 16 as our final configuration, as it achieves the highest performance while providing sufficient temporal expressiveness for complex video understanding tasks.

Table 3.

Top-K Selection Analysis