The lowest possible thermal conductivity of silicon-germanium (SiGe) bulk alloys achievable through alloy scattering, or the so-called alloy limit, is important to identify for thermoelectric applications. However, this limit remains a subject of contention as both experimentally-reported and theoretically-predicted values tend to be widely scattered and inconclusive. In this work, we present a possible explanation for these discrepancies by demonstrating that the thermal conductivity can vary significantly depending on the degree of randomness in the spatial arrangement of the constituent atoms. Our study suggests that the available experimental data, obtained from alloy samples synthesized using ball-milling techniques, and previous first-principles calculations, restricted by small supercell sizes, may not have accessed the alloy limit. We find that low-frequency anharmonic phonon modes can persist unless the spatial distribution of Si and Ge atoms is completely random at the atomic scale, in which case the lowest possible thermal conductivity may be achieved. Our theoretical analysis predicts that the alloy limit of SiGe could be around 1-2 W m-1 K-1 with an optimal composition around 25 at% Ge, which is substantially lower than previously reported values from experiments and first-principles calculations.