Optimizing Reservoir Operations with Reinforcement Learning: A Data-Driven Framework

Document Type : Original Article

Authors

1 Department of Civil Engineering, University of Mohaghegh Ardabili, Ardabil, Iran

2 Department of Water Engineering, Islamic Azad University, Garmsar, Iran

3 Faculty of Civil, Water & Environmental Engineering, Shahid Beheshti University, Tehran, Iran

Abstract

Effective reservoir management demands adaptive, data-driven strategies to optimize storage and release decisions while balancing multiple, often competing, operational objectives. This study investigates the application of Q-Learning, a model-free reinforcement learning (RL) algorithm, for optimizing reservoir releases under dynamic and uncertain hydrological conditions. Unlike conventional rule-based or offline optimization methods, the proposed RL approach continuously refines its release policy by learning from environmental feedback and observed states, enabling real-time adaptation without the need for a predefined system model. The framework is tested on the Dez Reservoir in Iran, a real-world case study characterized by significant inflow variability and seasonal water demand. Simulation results demonstrate that Q-Learning effectively manages operational complexity, maintaining storage within prescribed bounds and delivering release patterns closely aligned with demand. To benchmark performance, a simplified Ant Colony Optimization (ACO) model is implemented for comparison. While ACO shows moderate capability in deficit reduction, Q-Learning outperforms it in terms of constraint satisfaction and long-term feasibility. Findings highlight the strong potential of reinforcement learning to support intelligent, scalable, and robust decision-making in modern reservoir operation systems under uncertainty.

Keywords

Main Subjects

References

  1. Emami-Skardi, M. J., Shobeyri, G., Kerachian, R. Analysis of Stakeholder Relationships and Conflicts Using the Conflict Tree Approach. Iran-Water Resources Research, 2023; 18: 57–74. doi:20.1001.1.17352347.1401.18.4.4.8.
  2. Afshar, A., Emami Skardi, M. J., Jerani, F. Pond designing optimization using Multi-Objective ant colony algorithm and swat model. Journal of Environmental Science and Technology, 2015; 16 (Special issue): 121–132.
  3. Masoumi, F., Masoumzadeh, S., Zafari, N., Emami-Skardi, M. J. Optimal operation of single and multi-reservoir systems via hybrid shuffled grey wolf optimization algorithm (SGWO). Water Supply, 2021; 22: 1663–1675. doi:10.2166/ws.2021.326.
  4. Afshar, A., Emami Skardi, M. J., Masoumi, F. Optimizing water supply and hydropower reservoir operation rule curves: An imperialist competitive algorithm approach. Engineering Optimization, 2015; 47: 1208–1225. doi:10.1080/0305215X.2014.958732.
  5. Emami Skardi, M. J., Afshar, A., Saadatpour, M., Sandoval Solis, S. Hybrid ACO–ANN-Based Multi-objective Simulation–Optimization Model for Pollutant Load Control at Basin Scale. Environmental Modeling & Assessment, 2015; 20: 29–39. doi:10.1007/s10666-014-9413-7.
  6. Skardi, M. J. E., Afshar, A., Solis, S. S. Simulation-optimization model for non-point source pollution management in watersheds: Application of cooperative game theory. KSCE Journal of Civil Engineering, 2013; 17: 1232–1240. doi:10.1007/s12205-013-0077-7.
  7. Moradikian, S., Emami-Skardi, M. J., Kerachian, R. A distributed constraint multi-agent model for water and reclaimed wastewater allocation in urban areas: Application of a modified ADOPT algorithm. Journal of Environmental Management, 2022; 317: 115446. doi:10.1016/j.jenvman.2022.115446.
  8. Emamjomehzadeh, O., Kerachian, R., Emami-Skardi, M. J., Momeni, M. Combining urban metabolism and reinforcement learning concepts for sustainable water resources management: A nexus approach. Journal of Environmental Management, 2023; 329: 117046. doi:10.1016/j.jenvman.2022.117046.
  9. Masoumi, F., Masoumzadeh, S., Zafari, N., Javad Emami-Skardi, M. Optimum sanitary sewer network design using shuffled gray wolf optimizer. Journal of Pipeline Systems Engineering and Practice, 2021; 12: 04021055. doi:10.1061/(ASCE(PS.1949-1204.0000597.
  10. Tabari, M. M. R., Azadani, M. N., Kamgar, R. Development of operation multi-objective model of dam reservoir under conditions of temperature variation and loading using NSGA-II and DANN models: a case study of Karaj/Amir Kabir dam. Soft Computing, 2020; 24: 12469–12499. doi:10.1007/s00500-020-04686-1.
  11. Tabari, M. M. R., Safari, R. Development of water re-allocation policy under uncertainty conditions in the inflow to reservoir and demands parameters: a case study of Karaj AmirKabir dam. Soft Computing, 2023; 27: 6521–6547. doi:10.1007/s00500-023-07885-8.
  12. Eyni, A., Skardi, M. J. E., Kerachian, R. A regret-based behavioral model for shared water resources management: Application of the correlated equilibrium concept. Science of The Total Environment, 2021; 759: 143892. doi:10.1016/j.scitotenv.2020.143892.
  13. Emamjomehzadeh, O., Omidi, F., Kerachian, R., Emami-Skardi, M. J., Momeni, M. Water-energy-food-greenhouse gases nexus management in urban environments: A robust multi-agent decision-support system. Sustainable Cities and Society, 2024; 113: 105676. doi:10.1016/j.scs.2024.105676.
  14. Sharifian, H., Emami-Skardi, M. J., Behzadfar, M., Faizi, M. Water sensitive urban design (WSUD) approach for mitigating groundwater depletion in urban geography; through the lens of stakeholder and social network analysis. Water Supply, 2022; 22: 5833–5852. doi:10.2166/ws.2022.206.
  15. Li, C., Zhou, J., Ouyang, S., Ding, X., Chen, L. Improved decomposition–coordination and discrete differential dynamic programming for optimization of large-scale hydropower system. Energy Conversion and Management, 2014; 84: 363–373. doi:10.1016/j.enconman.2014.04.065.
  16. Zhao, T., Cai, X., Lei, X., Wang, H. Improved dynamic programming for reservoir operation optimization with a concave objective function. Journal of Water Resources Planning and Management, 2012; 138: 590–596. doi:10.1061/(ASCE)WR.1943-5452.0000205.
  17. Zeng, X., Hu, T., Cai, X., Zhou, Y., Wang, X. Improved dynamic programming for parallel reservoir system operation optimization. Advances in Water Resources, 2019; 131: 103373. doi:10.1016/j.advwatres.2019.07.003.
  18. Tegegne, G., Kim, Y.-O. Representing inflow uncertainty for the development of monthly reservoir operations using genetic algorithms. Journal of Hydrology, 2020; 586: 124876. doi:10.1016/j.jhydrol.2020.124876.
  19. Al-Aqeeli, Y. H., Mahmood Agha, O. M. A. Optimal Operation of Multi-reservoir System for Hydropower Production Using Particle Swarm Optimization Algorithm. Water Resources Management, 2020; 34: 3099–3112. doi:10.1007/s11269-020-02583-8.
  20. Kim, Y.-G., Sun, B.-Q., Kim, P., Jo, M.-B., Ri, T.-H., Pak, G.-H. A study on optimal operation of gate-controlled reservoir system for flood control based on PSO algorithm combined with rearrangement method of partial solution groups. Journal of Hydrology, 2021; 593: 125783. doi:10.1016/j.jhydrol.2020.125783.
  21. Ahmadianfar, I., Kheyrandish, A., Jamei, M., Gharabaghi, B. Optimizing operating rules for multi-reservoir hydropower generation systems: An adaptive hybrid differential evolution algorithm. Renewable Energy, 2021; 167: 774–790. doi:10.1016/j.renene.2020.11.152.
  22. Emami, M., Nazif, S., Mousavi, S.-F., Karami, H., Daccache, A. A hybrid constrained coral reefs optimization algorithm with machine learning for optimizing multi-reservoir systems operation. Journal of Environmental Management, 2021; 286: 112250. doi:10.1016/j.jenvman.2021.112250.
  23. Moeini, R., Babaei, M. Hybrid SVM-CIPSO methods for optimal operation of reservoir considering unknown future condition. Applied Soft Computing, 2020; 95: 106572. doi:10.1016/j.asoc.2020.106572.
  24. Ehteram, M., Banadkooki, F. B., Fai, C. M., Moslemzadeh, M., Sapitang, M., Ahmed, A. N., Irwan, D., El-Shafie, A. Optimal operation of multi-reservoir systems for increasing power generation using a seagull optimization algorithm and heading policy. Energy Reports, 2021; 7: 3703–3725. doi:10.1016/j.egyr.2021.06.008.
  25. Chong, K. L., Lai, S. H., Ahmed, A. N., Wan Jaafar, W. Z., El-Shafie, A. Optimization of hydropower reservoir operation based on hedging policy using Jaya algorithm. Applied Soft Computing, 2021; 106: 107325. doi:10.1016/j.asoc.2021.107325.
  26. Paliwal, V., Ghare, A. D., Mirajkar, A. B., Bokde, N. D., Feijoo Lorenzo, A. E. Computer modeling for the operation optimization of mula reservoir, Upper Godavari Basin, India, Using the Jaya Algorithm. Sustainability, 2019; 12: 84. doi:10.3390/su12010084.
  27. Latif, S. D., Marhain, S., Hossain, M. S., Ahmed, A. N., Sherif, M., Sefelnasr, A., El-Shafie, A. Optimizing the operation release policy using charged system search algorithm: a case study of Klang Gates Dam, Malaysia. Sustainability, 2021; 13: 5900. doi:10.3390/su13115900.
  28. Zhang, Z., Qin, H., Yao, L., Liu, Y., Jiang, Z., Feng, Z., Ouyang, S. Improved Multi-objective Moth-flame Optimization Algorithm based on R-domination for cascade reservoirs operation. Journal of Hydrology, 2020; 581: 124431. doi:10.1016/j.jhydrol.2019.124431.
  29. Feng, Z.-k., Liu, S., Niu, W.-j., Li, B.-j., Wang, W.-c., Luo, B., Miao, S.-m. A modified sine cosine algorithm for accurate global optimization of numerical functions and multiple hydropower reservoirs operation. Knowledge-Based Systems, 2020; 208: 106461. doi:10.1016/j.knosys.2020.106461.
  30. Yaseen, Z. M., Allawi, M. F., Karami, H., Ehteram, M., Farzin, S., Ahmed, A. N., Koting, S. B., Mohd, N. S., Jaafar, W. Z. B., Afan, H. A., El-Shafie, A. A hybrid bat–swarm algorithm for optimizing dam and reservoir operation. Neural Computing and Applications, 2019; 31: 8807–8821. doi:10.1007/s00521-018-3952-9.
  31. Liu, D., Huang, Q., Yang, Y., Liu, D., Wei, X. Bi-objective algorithm based on NSGA-II framework to optimize reservoirs operation. Journal of Hydrology, 2020; 585: 124830. doi:10.1016/j.jhydrol.2020.124830.
  32. Feng, Z.-k., Niu, W.-j., Zhang, R., Wang, S., Cheng, C.-t. Operation rule derivation of hydropower reservoir by k-means clustering method and extreme learning machine based on particle swarm optimization. Journal of Hydrology, 2019; 576: 229–238. doi:10.1016/j.jhydrol.2019.06.045.
  33. Meng, X., Chang, J., Wang, X., Wang, Y. Multi-objective hydropower station operation using an improved cuckoo search algorithm. Energy, 2019; 168: 425–439. doi:10.1016/j.energy.2018.11.096.
  34. Zhang, R., Zhou, J., Ouyang, S., Wang, X., Zhang, H. Optimal operation of multi-reservoir system by multi-elite guide particle swarm optimization. International Journal of Electrical Power & Energy Systems, 2013; 48: 58–68. doi:10.1016/j.ijepes.2012.11.031.
  35. Raso, L., Bader, J.-C., Weijs, S. Reservoir operation optimized for hydropower production reduces conflict with traditional water uses in the Senegal River. Journal of Water Resources Planning and Management, 2020; 146: 05020003. doi:10.1061/(ASCE)WR.1943-5452.0001076.
  36. Luo, J., Qi, Y., Xie, J., Zhang, X. A hybrid multi-objective PSO–EDA algorithm for reservoir flood control operation. Applied Soft Computing, 2015; 34: 526–538. doi:10.1016/j.asoc.2015.05.036.
  37. Niu, W.-j., Feng, Z.-k., Liu, S. Multi-strategy gravitational search algorithm for constrained global optimization in coordinative operation of multiple hydropower reservoirs and solar photovoltaic power plants. Applied Soft Computing, 2021; 107: 107315. doi:10.1016/j.asoc.2021.107315.
  38. Li, F.-F., Qiu, J. Multi-objective optimization for integrated hydro–photovoltaic power system. Applied Energy, 2016; 167: 377–384. doi:10.1016/j.apenergy.2015.09.018.
  39. Bai, T., Wei, J., Chang, F.-J., Yang, W., Huang, Q. Optimize multi-objective transformation rules of water-sediment regulation for cascade reservoirs in the Upper Yellow River of China. Journal of Hydrology, 2019; 577: 123987. doi:10.1016/j.jhydrol.2019.123987.
  40. Kurek, W., Ostfeld, A. Multi-objective optimization of water quality, pumps operation, and storage sizing of water distribution systems. Journal of Environmental Management, 2013; 115: 189–197. doi:10.1016/j.jenvman.2012.11.030.
  41. Allawi, M. F., Jaafar, O., Mohamad Hamzah, F., Koting, S. B., Mohd, N. S. B., El-Shafie, A. Forecasting hydrological parameters for reservoir system utilizing artificial intelligent models and exploring their influence on operation performance. Knowledge-Based Systems, 2019; 163: 907–926. doi:10.1016/j.knosys.2018.10.013.
  42. Asadieh, B., Afshar, A. Optimization of water-supply and hydropower reservoir operation using the charged system search algorithm. Hydrology, 2019; 6: 5. doi:10.3390/hydrology6010005.
  43. He, Y., Xu, Q., Yang, S., Liao, L. Reservoir flood control operation based on chaotic particle swarm optimization algorithm. Applied Mathematical Modelling, 2014; 38: 4480–4492. doi:10.1016/j.apm.2014.02.030.
  44. Turgut, M. S., Turgut, O. E., Afan, H. A., El-Shafie, A. A novel Master–Slave optimization algorithm for generating an optimal release policy in case of reservoir operation. Journal of Hydrology, 2019; 577: 123959. doi:10.1016/j.jhydrol.2019.123959.
  45. Skardi, M. J. E., Kerachian, R., Abdolhay, A. Water and treated wastewater allocation in urban areas considering social attachments. Journal of Hydrology, 2020; 585: 124757. doi:10.1016/j.jhydrol.2020.124757.
Civil Engineering and Applied Solutions
Volume 1, Issue 4
September 2025
Pages 56-67

Files

History

  • Receive Date: 28 July 2025
  • Revise Date: 03 August 2025
  • Accept Date: 20 August 2025
  • First Publish Date: 20 August 2025

Share

How to cite

Statistics