DS-CAE: A Dual-Stream Cross-Attentive Autoencoder for Robust and Cluster-Aware Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) enhances language models by incorporating external knowledge at inference time. However, its performance is highly sensitive to the quality of retrieved content, which often includes noisy or irrelevant distractors. Conventional latent representations for clustering-based retrieval are often poorly structured and misaligned with generation objectives. We propose DS-CAE, a Dual-Stream Cross-Attentive Autoencoder that jointly encodes global and local semantics via Transformer and BiLSTM encoders. We fuse them with bidirectional cross-Attention and token-wise gating for context-Aware integration. We improve retrieval with a composite loss (reconstruction + adaptive-margin triplet) and GMM filtering based on reconstruction loss and cluster distance. Experiments on Natural Questions, WebQuestions, and CuratedTREC show that DS-CAE outperforms previous RAG models on CuratedTREC by up to 6.8% and performs competitively on NQ with a compact 171M-parameter model. We validate each component's impact on cluster-Aware retrieval through ablations.