Assessing Grid Penalized Reinforcement Learning for Renewable Energy Management of Power-to-X Integrated With Intermediate Storage

This research explores the deep reinforcement learning (DRL) based planning strategies of power-to-X (PtX) systems under uncertainties of renewable and price through a detailed case study and comparative analysis of system planning. A DRL-based hourly planning model is proposed to minimize operational costs for a PtX system, incorporating a hybrid energy storage system. The model employs a grid-penalized reward function to manage grid power usage while accounting for temporal uncertainties in renewable and grid prices. To analyze the DRL model's planning strategies, it is compared to a general rule-based model across varying spatial and temporal uncertainties using real-world data from national (France) areas. The results show that the DRL-based planning approach consistently outperforms the rule-based model, achieving 1,360.12% higher monthly profits in the national area, though with a relatively lower renewable energy penetration (REP). However, sensitivity analysis reveals that increasing the grid penalty level effectively reduces the gap in REP while sustaining higher profitability. This comparative analysis is the first to quantitatively reveal the planning strategies of a DRL-based PtX system, highlighting its effectiveness in reducing grid power overuse while maintaining higher profitability in system planning.