Using data from the Korean Genome and Epidemiology Study, we identified weight-change patterns during midlife using a group-based trajectory model, and evaluated their associations with the incidence of cardiovascular disease (CVD). At baseline, there were 8774 CVD-free participants. Group-based modeling was used to analyze patterns of weight change over about 16 years. Using multiple model, we evaluated the association between weight-change patterns and CVD risk. During the follow-up period, 741 new CVD cases were identified. The weight-change patterns were characterized as ‘gradual weight gain’, ‘stable weight’, ‘slight weight loss’, and ‘gradual weight loss’. The association between weight-change patterns and CVD risk differed depending on the level of physical activity (PA) at baseline (pinteraction < 0.05). Compared with the stable-weight group, the risk of all CVD (HR 2.5, 95% CI 1.5–4.3) and non-fatal CVD (HR 2.8, 95% CI 1.6–4.9) among the gradual-weight-loss group was apparent in the lowest PA quartile. In addition, on average, a decrease in skeletal-muscle-mass (SMM) levels was observed during the follow-up period, but the decrease in SMM in the gradual-weight-loss group was greater than in the gradual-weight-gain group. Our findings show that gradual weight loss was associated with CVD risk, which was dependent on PA levels.