Whole brain radiotherapy (WBRT) is a mainstay of the treatment for brain metastases (BM). We evaluated prognostic factors in colorectal cancer (CRC) patients undergoing WBRT for BM. The medical records of 106 CRC patients undergoing WBRT for BM between 2000 and 2014 at three institutions were reviewed. Patient and tumor factors were analyzed to identify the prognostic factors for overall survival (OS) calculated from the date of BM diagnosis to the date of death or last follow-up. Surgical resection of BM was performed in six patients. The dose of WBRT was 30 Gy, and boost radiotherapy or stereotactic radiosurgery (8–23 Gy) was given to 15 patients. Systemic therapy for BM was administered in one patient before WBRT and 26 patients after WBRT. The median follow-up time was 3.9 months (range, 0.4–114.1 months). The median OS time was 3.9 months, and the 1-year OS rate was 18.2%. Older age (>65 years), multiple BM (≥3), elevated level of carcinoembryonic antigen (CEA, >5 ng/ml) at BM diagnosis, and extracranial metastases were adverse prognostic factors for OS. Patient with 0–1 factor showed better OS (at 1 year, 76.9%) than patients with 2 factors (16.7%) or 3–4 factors (4.2%; p < 0.001). In conclusion, we evaluated age, the number of BM, CEA level, and extracranial metastases as the prognostic factors for OS in CRC patients undergoing WBRT. Our result might be useful to develop prognostic models predicting survival for patients whom WBRT is intended for.