Efficient training of large language models on distributed infrastructures: a survey

The proliferation of Large Language Models (LLMs) like GPT and LLaMA has profoundly transformed the artificial intelligence landscape, yet their training demands substantial computational resources, presenting significant challenges in scalability, efficiency, and system reliability. This research direction provides guidance on understanding the recent advancements in distributed training systems specifically tailored for LLMs. It encompasses innovations across various infrastructure components, including specialized AI accelerators, high-throughput networking solutions, optimized storage architectures, and intelligent scheduling algorithms designed for large-scale deployments. Furthermore, the scope includes an in-depth analysis of diverse parallelism strategies, such as data, model, and pipeline parallelism, alongside critical optimizations for computation, inter-node communication, and memory management within distributed training environments. Emphasis

is also placed on methodologies for maintaining system reliability and fault tolerance during extended training durations. This topic further explores the inherent limitations of traditional digital circuit-based computing systems in meeting the escalating computational demands of LLMs, investigating the potential of emerging solutions like optical computing and optical networks. The objective is to offer a structured approach for M.Tech students to explore and contribute to the enhancement of LLM training systems through research, simulation, or comparative analysis.

Message: Hi FE, I need help with “Efficient training of large language models on distributed infrastructures: a survey” in “Computer Science & Engineering”