We investigate a depth-based refocusing method using four-dimensional (4D) light field data, which can reduce directional aliasing artifacts in a refocused image. Unlike conventional filtering-based methods, the proposed method estimates the amount of aliasing artifacts using the disparity information between two neighboring views. It then applies an exact smoothing operation to the refocused image in order to remove the aliasing artifacts. By doing that, the proposed method is able to generate a smoothly blurred image in the out-of-focus region and a sharp image in the focused region. In addition, as the proposed method performs a smoothing operation on the refocused image, it does not create virtual views, which often requires an extremely large amount of computational resources. In both simulation and experiment, the proposed method shows outstanding performance compared to conventional methods.