PhD Dissertation Proposal Defense: Kunjal Panchal, Advancing Machine Learning for Resource-Constrained Environments

Content

Speaker: Kunjal Panchal 
Advisor: Hui Guan 

Abstract: Machine learning (ML) on resource-constrained devices enables real-time, low-latency processing without reliance on server connectivity, which is critical for privacy, cost efficiency, and applications in remote or bandwidth-limited environments. However, resource-constrained devices like IoT devices and commodity mobile phones suffer from lack of sufficient data and compute for training. Federated Learning (FL) addresses these issues by enabling devices to collaboratively train a shared ML model using their individual data and compute resources.

Despite its promise, FL faces challenges such as (a) data heterogeneity across devices, (b) temporal data heterogeneity on each device, and (c) ML models exceeding the memory capacity of resource-constrained devices. This dissertation proposes works addressing the heterogeneity and memory-consumption challenges through (i) Flow (NeurIPS 2023): Per-instance dynamic personalization through a routing mechanism for devices with heterogeneous data, (ii) Flash (ICML 2023): Client-side drift detection and server-side drift-aware optimization for temporal data heterogeneity, and (iii) Spry (NeurIPS 2024): Using forward-mode auto differentiation, an alternative to backpropagation, to compute gradients with 7x less memory consumption. Ongoing and future work will focus on optimizing forward-mode auto differentiation to make it more practical to use in resource-constrained devices. It includes (1) improving the convergence speed and converged accuracy to be closer to backpropagation via novel optimizer designs, and (2) further reducing memory consumptions of optimizers via low-rank perturbations. Together, these works address critical challenges in enabling machine learning on resource-constrained devices by tackling data and temporal heterogeneity, as well as memory limitations.

Join the Zoom