GPU Accelerated Optimization Algorithms

GPU Accelerated Optimization Algorithms
A Proposal of Undergraduate Research Project by Zhiliang Xu

Project Narrative. Training a machine learning model is essentially an optimization problem. Optimization algorithm is utilized to find values of model hyperparameters that minimize model errors when mapping model inputs to model outputs. This typically is formulated as minimizing a so-called loss function. The goal of this project is two-fold: 1) we will explore a few widely used optimization algorithms such as gradient descent and stochastic gradient descent algorithms; we will investigate how they are used in machine learning models, and the mathematics behind them; 2) we will implement some of these algorithms using graphic processing units (GPUs) to achieve parallel efficiency. The main deliverables at the end of the summer are project report and computer codes. The work can potentially be presented/published.

Outline of Student Work. The first two - three weeks are literature search and reading. Materials include numerical analysis theories, aforementioned optimization algorithms. The next two - three weeks are for learning basics of programming on GPUs. We will be exploring CUDA (Compute Unified Device Architecture) programming model developed by NVIDIA. Rest of the summer research time is for implementing optimization algorithms and developing the project report. Student working on this project is expected to meet with the mentor every week (can be flexible).