Background: The veins from the lower rectum drain into the systemic venous system, while those from other parts of the colon drain into the portal venous system. The aim of this study was to investigate recurrence pattern and survival according to the anatomical differences in patients with colorectal liver metastases (CRLM). Methods: From October 1994 to December 2009, synchronous CRLM patients who underwent surgery were identified from our prospectively collected database. The patients were excluded if there had been extrahepatic metastases. The patients were divided into two groups according to the location of the primary colorectal cancer: lower rectal cancer (group 1) and upper rectal or colon cancer (group 2). The recurrence patterns and survival were investigated. Results: A total of 316 patients were included: 53 patients in group 1 and 263 patients in group 2. After a median follow-up of 37 months, the extrahepatic recurrence curve of group 1 was superior to that of group 2 (P < 0.001), although there was no difference between the hepatic recurrence curves (P = 0.93). The disease-free and overall survival curves of group 1 were inferior to those of group 2 (P = 0.004) (P < 0.001). Lower rectal cancer was a significant risk factor for extrahepatic recurrence in Cox proportional hazard model analysis (hazard ratio = 1.7, P = 0.04). Conclusions: The extrahepatic recurrence rate is high in lower rectal cancer patients after surgical treatment for synchronous CRLM.