Since energy expenditure (EE) has been proven an effective representation of physiological demands, field EE measurement enables us to assess worker's physical demands from ongoing construction tasks. However, because EE measurement requires special facilities and equipment, collecting data in the field remains challenging. One possible solution is to use heart rate (HR) to estimate EE according to a linear relationship between HR and EE. Recent advancements in a wristband-type activity tracker can potentially open new doors for field EE measurement, yet because its capacity for measurement is limited to aerobic free-living activities (e.g., walking), EE measurement may not be accurate for physically demanding construction works. To examine the potential of continuously measuring construction workers' EE from an activity tracker, we identified the following challenges: (1) need for different HR-EE relations for diverse tasks; (2) application of corresponding HR-EE relationship to such tasks and (3) accurate HR monitoring. To overcome these challenges, our in-depth investigation indicated that a wristband-type activity tracker has the potential to achieve reliable field EE measurement through automatic action recognition using an embedded accelerometer, and applying HR-EE relationships for corresponding actions with acceptable HR monitoring accuracy. By addressing these challenges, a proposed field EE measurement approach is expected to provide in-depth understanding and continuous monitoring of worker's physical demands from construction tasks.