Sparse Neural Networks 분야의 세계적인 전문가인University of Calgary의 Yani Ioannou 교수님을 모시고 아래와 같이 세미나를 개최하오니, 관심있는 많은 분들의 참여 부탁 드립니다.

*일시: 9월 10일(수) 오후 1시 30분

*장소: 반도체관 400112호

*연사: Prof. Yani Ioannou (University of Calgary, Canada)

*Talk Title: Training Structured Sparse Neural Networks

Abstract

The challenge in training sparse neural networks is to achieve both high accuracy and practical hardware acceleration. Unstructured sparsity often yields good performance but is hard to speed up, while traditional structured sparsity can hurt performance. Our International Conference in Learning Representations (ICLR) 2024 paper, “Dynamic Sparse Training with Structured Sparsity” introduces Structured RigL (SRigL) to address this by dynamically learning hardware-friendly sparse weight representations without sacrificing accuracy.

SRigL successfully learns a combination of fine-grained N:M structured sparsity (constant fan-in) and neuron-level sparsity (neuron ablation) dynamically from a sparse initialization. The explicit integration of neuron ablation, a behavior implicitly learned by unstructured DST methods at high sparsities, is crucial for SRigL to match the generalization performance of dense and unstructured sparse models, even at extreme sparsities (up to 99%). The learned structured sparsity enables a “condensed sparse representation,” which translates to significant real-world inference speedups on commodity CPUs and GPUs outperforming unstructured sparse formats in many practical scenarios. SRigL demonstrates a viable path to train sparse neural networks that are both highly accurate and practically efficient by learning structured masks that can exploit hardware acceleration, bridging the gap between unstructured DST performance and structured sparsity acceleration.

Biography

Dr. Yani Ioannou is an Assistant Professor and Schulich Research Chair at the University of Calgary in the Department of Electrical and Software Engineering of the Schulich School of Engineering, and leads the Calgary Machine Learning Lab. He was previously a Postdoctoral Research Fellow at the Vector Institute and University of Guelph, working with Prof. Graham Taylor, and a Visiting Researcher at Google Brain Toronto with Prof. Geoffrey Hinton (Nobel/Turing Award Winner) and Dr. Sharam Izadi (Google AR Core). Yani completed his PhD at the University of Cambridge in 2018 supported by a Microsoft Research Ph.D. Scholarship, where he was supervised by Prof. Roberto Cipolla and Dr. Antonio Criminisi. Dr. Ioannou’s research focuses on efficient and trustworthy deep learning, with a specific focus on sparse neural network training and inference.

*Host: 고종환 교수 (전자전기컴퓨터공학과)