High-concentration doping of silicon (Si)-based materials is an effective way to improve their thermoelectric efficiency via thermal conductivity (κ) reduction as well as thermopower and electrical conductivity enhancement. Beyond the solubility limit, a large portion of dopant atoms may undergo clustering mediated primarily by point defects, which may in turn significantly alter the physical properties of host materials. In this work, we investigate the effect of Arsenic-vacancy (AsV) complex formation on κ in heavily As-doped Si using molecular dynamics simulations. The simulation results clearly demonstrate that the presence of AsV complexes, particularly As4V which is the most stable one, may result in a substantially lower κ compared to the case where all As atoms remain substitutional and electrically active. Further analysis reveals that the central vacancy in As4V causes Si lattice softening and thus results in the reduction of phonon group velocity. Our findings highlight the significant effect of dopant clustering on κ and also provide some guidance on how to manipulate Si-based materials to improve their thermoelectric performance via doping combined with defect engineering.