Treffer: Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

Title:
Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence
Authors:
Ejarque Artigas, Jorge, Badia Sala, Rosa Maria, Becerra Fontal, Yolanda, Rodrigo Berlín, Julián, Folch Duran, Arnau, Lordan Gomis, Francesc, Monterrubio Velasco, Marisol, de la Puente, Josep, Queralt Calafat, Anna, Rodríguez Rodríguez, Juan E., Rossi, Riccardo
Contributors:
Universitat Politècnica de Catalunya. Departament d'Arquitectura de Computadors, Universitat Politècnica de Catalunya. Departament d'Enginyeria de Serveis i Sistemes d'Informació, Universitat Politècnica de Catalunya. Departament d'Enginyeria Civil i Ambiental, Barcelona Supercomputing Center, Centre Internacional de Mètodes Numèrics en Enginyeria, Universitat Politècnica de Catalunya. CAP - Grup de Computació d'Altes Prestacions, Universitat Politècnica de Catalunya. inSSIDE - integrated Software, Service, Information and Data Engineering, Universitat Politècnica de Catalunya. RMEE - Grup de Resistència de Materials i Estructures en l'Enginyeria
Publisher Information:
Elsevier
Publication Year:
2022
Collection:
Universitat Politècnica de Catalunya, BarcelonaTech: UPCommons - Global access to UPC knowledge
Subject Terms:
Àrees temàtiques de la UPC::Informàtica::Arquitectura de computadors, High performance computing, Electronic data processing -- Distributed processing, Parallel programming (Computer science), Distributed computing, HPC-DA-AI convergence, Workflow development, Workflow orchestration, Càlcul intensiu (Informàtica), Processament distribuït de dades, Programació en paral·lel (Informàtica)
Document Type:
Fachzeitschrift article in journal/newspaper
File Description:
16 p.; application/pdf
Language:
English
Relation:
https://www.sciencedirect.com/science/article/abs/pii/S0167739X22001364; info:eu-repo/grantAgreement/EC/H2020/955558/EU/Enabling dynamic and Intelligent workflows in the future EuroHPCecosystem/eFlows4HPC; https://arxiv.org/abs/2204.09287; http://hdl.handle.net/2117/367940
DOI:
10.1016/j.future.2022.04.014
Availability:
http://hdl.handle.net/2117/367940
https://arxiv.org/abs/2204.09287
https://doi.org/10.1016/j.future.2022.04.014
Rights:
Attribution-NonCommercial-NoDerivatives 4.0 International ; http://creativecommons.org/licenses/by-nc-nd/4.0/ ; Open Access
Accession Number:
edsbas.ADFCC2DC
Database:
BASE

Weitere Informationen

The evolution of High-Performance Computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows; and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project. ; This work has received funding from the European HighPerformance Computing Joint Undertaking (JU) under grant agreement No 955558. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Spain, Germany, France, Italy, Poland, Switzerland and Norway. In Spain, it has received complementary funding from MCIN/AEI/10.13039/501100011033, Spain and the European Union NextGenerationEU/PRTR (contracts PCI2021-121957, PCI2021-121931, ...