LIGHTWEIGHT TEMPORAL CONTEXTUAL FINE-TUNING METHOD OF LARGE MULTIMODAL MODEL FOR VIDEO MOMENT RETRIEVAL

Video moment retrieval (VMR) tasks require a comprehensive understanding of the video-language features of an input video, based on large multimodal models (LMMs). In this paper, we introduce temporal contextual prompts and provide contextual information into the LMM model to improve the generalization ability of the VMR. We integrate temporal contextual prompts into conventional prompts and obtain integrated tokens through an embedding module. The temporal contextual prompts are transformed into temporal conditional tokens, and these are fine-tuned as the embedding representation of the temporal correlation between the video and the given query text. In addition, we quantize the model and apply LoRA to demonstrate efficient learning for limited resources. Experimental results on Charades-STA show improvements of 3.78 percent in mIoU and 0.85 percent in [email protected] over state-of-the-art performance.