Multi-Agent Microservice Architecture

Abstract

Microservice architecture is an integral part of the distributed software systems that require continuous scaling and independent deployment of all their elements. The microservice advantages could significantly increase efficiency of the modern web applications and open up new opportunities in the business development. However, dynamic variability of the modern Internet services, evolution in the user needs, as well as various external factors could negate advantages of the microservice architecture. One of the promising methods in adaptive resource management of the distributed software systems includes the machine learning algorithms, especially the deep reinforcement learning algorithms. The paper considers integration of the microservice architecture and the multi-agent reinforcement learning. Combining these approaches makes it possible to optimize the web applications operation in the non-stationary environments allowing the system to adapt to alterations and find the optimal solutions. The paper provides results of learning a classical multi-agent independent Q-learning algorithm in the road route selection service based on the current weather conditions. To evaluate the system efficiency, additional service quality parameters were developed and introduced making it possible to fully evaluate potential of integrating the microservice architecture and the multi-agent learning in solving complex problems in the dynamic environments

The work was performed with support by the Ministry of Science and Higher Education of the Russian Federation within the framework of the State Task (project no. FSFN-2024-0059)

Please cite this article in English as:

Morgunov E.F., Alfimtsev A.N. Multi-agent microservice architecture. Herald of the Bauman Moscow State Technical University, Series Instrument Engineering, 2025, no. 2 (151), pp. 78--101 (in Russ.). EDN: TPNAZP

References

[1] Nadareishvili I., Mitra R., McLarty M., et al. Microservice architecture. Sebastopol, O’Reilly Media, 2016.

[2] Thones J. Microservices. IEEE Softw., 2015, vol. 32, no. 1, p. 116. DOI: https://doi.org/10.1109/MS.2015.11

[3] Blinowski G., Ojdowska A., Przybylek A. Monolithic vs. microservice architecture: a performance and scalability evaluation. IEEE Access, 2022, vol. 10, pp. 20357--20374. DOI: https://doi.org/10.1109/ACCESS.2022.3152803

[4] Hasselbring W., Steinacker G. Microservice architectures for scalability, agility and reliability in e-commerce. IEEE ICSAW, 2017, pp. 243--246. DOI: https://doi.org/10.1109/ICSAW.2017.11

[5] Camero A., Alba E. Smart City and information technology: a review. Cities, 2019, vol. 93, pp. 84--94. DOI: https://doi.org/10.1016/j.cities.2019.04.014

[6] Jordan M.I., Mitchell T.M. Machine learning: trends, perspectives, and prospects. Science, 2015, vol. 349, no. 6245, pp. 255--260. DOI: https://doi.org/10.1126/science.aaa8415

[7] Sutton R.S., Barto A.G. Reinforcement learning. An introduction. London, MIT Press Cambridge, 2018.

[8] Canese L., Cardarilli G.C., Di Nunzio L., et al. Multi-agent reinforcement learning: a review of challenges and applications. Appl. Sc., 2021, vol. 11, no. 11, art. 4948. DOI: https://doi.org/10.3390/app11114948

[9] Lee D., He N., Kamalaruban P., et al. Optimization for reinforcement learning: from a single agent to cooperative agents. IEEE Signal Process. Mag., 2020, vol. 37, no. 3, pp. 123--135. DOI: https://doi.org/10.1109/MSP.2020.2976000

[10] Putta P., Mills E., Garg N., et al. Agent Q: advanced reasoning and learning for auto-nomous AI agents. arXiv:2408.07199. DOI: https://doi.org/10.48550/arXiv.2408.07199

[11] Niknejad N., Ismail W., Ghani I., et al. Understanding Service-Oriented Architecture (SOA): a systematic literature review and directions for further investigation. Inf. Syst., 2020, vol. 91, art. 101491. DOI: https://doi.org/10.1016/j.is.2020.101491

[12] Wang H., Wang X., Hu X., et al. A multi-agent reinforcement learning approach to dynamic service composition. Inf. Sc., 2016, vol. 363, pp. 96--119. DOI: https://doi.org/10.1016/j.ins.2016.05.002

[13] Wang H., Gu M., Yu Q., et al. Adaptive and large-scale service composition based on deep reinforcement learning. Knowl.-Based Syst., 2019, vol. 180, no. 10, pp. 75--90. DOI: https://doi.org/10.1016/j.knosys.2019.05.020

[14] Wang H., Hu X., Yu Q., et al. Integrating reinforcement learning and skyline computing for adaptive service composition. Inf. Sc., 2020, vol. 519, pp. 141--160. DOI: https://doi.org/10.1016/j.ins.2020.01.039

[15] Collier R., O’Neill E., Lillis D., et al. MAMS: multi-agent microservices. WWW’19, 2019, pp. 655--662. DOI: https://doi.org/10.1145/3308560.3316509

[16] O’Neill E., Lillis D., O’Hare G.M., et al. Delivering multi-agent MicroServices using CArtAgO. In: Engineering multi-agent systems. Cham, Springer International Publishing, 2020, pp. 1--20. DOI: https://doi.org/10.1007/978-3-030-66534-0_1

[17] Zhu Y., Zhan Y., Huang X., et al. OFCOURSE: a multi-agent reinforcement learning environment for order fulfillment. NeurIPS, 2024, vol. 36. URL: https://proceedings.nips.cc/paper_files/paper/2023/hash/6d0cfc5db3feeabf6762129ba91bd3a1-Abstract-Datasets_and_Benchmarks.html (дата обращения: 15.02.2025).

[18] Newman S. Building microservices. Sebastopol, O’Reilly Media, 2021.

[19] Li S., Zhang H., Jia Z., et al. Understanding and addressing quality attributes of microservices architecture: a systematic literature review. Inf. Softw. Technol., 2021, vol. 131, art. 106449. DOI: https://doi.org/10.1016/j.infsof.2020.106449

[20] Velepucha V., Flores P. A survey on microservices architecture: principles, patterns and migration challenges. IEEE Access, 2023, vol. 11, pp. 88339--88358. DOI: https://doi.org/10.1109/ACCESS.2023.3305687

[21] Newman S. Monolith to microservices. Sebastopol, O’Reilly Media, 2019.

[22] Waseem M., Liang P., Shahin M. A systematic mapping study on microservices architecture in devops. J. Syst. Softw., 2020, vol. 170, art. 110798. DOI: https://doi.org/10.1016/j.jss.2020.110798

[23] Leroy P., Morato P.G., Pisane J., et al. IMP-MARL: a suite of environments for large-scale infrastructure management planning via MARL. NeurIPS, 2024. URL: https://proceedings.neurips.cc/paper_files/paper/2023/hash/a7a7c0c92f195cce85f99768621ac6c0-Abstract-Datasets_and_Benchmarks.html (дата обращения: 15.02.2025).

[24] Bacchiani L., Bravetti M., Giallorenzo S., et al. Microservice dynamic architecture-level deployment orchestration. In: Coordination Models and Languages. Cham, Springer Nature, 2021, pp. 257--275. DOI: https://doi.org/10.1007/978-3-030-78142-2_16

[25] Vinyals O., Ewalds T., Bartunov S., et al. Starcraft II: a new challenge for reinforcement learning. arXiv:1708.04782. DOI: https://doi.org/10.48550/arXiv.1708.04782

[26] Terry J., Black B., Grammel N., et al. Pettingzoo: gym for multi-agent reinforcement learning. NeurIPS, 2021, vol. 34, pp. 15032--15043.

[27] Fan J., Wang Z., Xie Y., et al. A theoretical analysis of deep Q-learning. Proc. 2nd Conf. on Learning for Dynamics and Control, 2020, pp. 486--489

[28] Bolshakov V.E., Alfimtsev A.N. Hierarchical method for cooperative multiagent reinforcement learning in Markov decision processes. Dokl. Math., 2023, vol. 108, no. 2S, pp. S382--S392. DOI: https://doi.org/10.1134/S1064562423701132

[29] Ozdaglar A., Sayin M.O., Zhang K. Independent learning in stochastic games. In: International Congress of Mathematicians, 2021, pp. 5340--5373. DOI: https://doi.org/10.4171/icm2022/152

[30] Zhang Z., Yang J., Zha H. Integrating independent and centralized multi-agent reinforcement learning for traffic signal network optimization. arXiv:1909.10651. DOI: https://doi.org/10.48550/arXiv.1909.10651

[31] Morgunov E.F., Alfimtsev A.N. The "stag hunt" social dilemma in multi-agent reinforcement learning. REEPE, 2024. DOI: https://doi.org/10.1109/REEPE60449.2024.10479770

[32] Zheng H., Zhao W., Yang J., et al. QoS analysis for web service compositions with complex structures. IEEE Trans. Serv. Comput., 2013, vol. 6, no. 3, pp. 373--386. DOI: https://doi.org/10.1109/TSC.2012.7

[33] Kuba J.G., Feng X., Ding S., et al. Heterogeneous-agent mirror learning: a continuum of solutions to cooperative marl. arXiv:2208.01682. DOI: https://doi.org/10.48550/2208.01682

[34] Leibo J.Z., Zambaldi V., Lanctot M., et al. Multi-agent reinforcement learning in sequential social dilemmas. arXiv:1702.03037. URL: https://doi.org/10.48550/arXiv.1702.03037