To meet the demand of higher realism, a high number of particles are used for particle-based fluid simulations, resulting in various out-of-core issues. In this paper, we present an out-of-core proximity computation, especially, e-Nearest Neighbor (e-NN) search, commonly used for particle-based fluid simulations, to handle such big data sets consisting of tens of millions of particles. Specifically, we identify a maximal work set that a GPU can process efficiently in an in-core mode. As a main technical component, we compute a memory footprint for processing a given work set based on our expectation model of the number of neighbors of particles. Our method can naturally utilize heterogeneous computing resources such as CPUs and GPUs, and has been applied to large-scale fluid simulations based on smoothed particle hydrodynamics. We have demonstrated that our method handles up to 65 M particles and processes up to 15 M ε-NN queries per second by using two CPUs and a GPU, which has only 3 GB video memory. This result is up to 51 × higher performance than a single CPU-core version for the out-of-core case. This high performance for large-scale data given a limited video memory space is achieved mainly thanks to the high accuracy of our memory estimation method.