Recent advances in generative models using deep neural networks have enabled scalable modeling of complex, high-dimensional data including images, videos, text and audio and have been transformative for science and industry. In this course, we will study the foundations and learning algorithms for deep generative models, including diffusion models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flow models, energy-based models, and score-based models. Part of the course will be devoted to teaching different approaches to generative modeling and the other part will be devoted to reading and discussing most recent papers about algorithms and applications of deep generative models in image and video understanding, graph mining and natural language modeling and robotics.

Basic knowledge about machine learning and probability as well as programming in Python.

1. Deep Generative Models

• Introduction and Background

• Autoregressive Models

• Variational Auto-Encoders (VAEs)

• Diffusion Models for Continuous, Discrete and Graph Data

• Generative Adversarial Networks (GANs)

• Normalizing Flows

• Energy Based Models

• Score Based Models

• Evaluating Generative Models

2. Paper Presentations on Deep Generative Models: This part of the course involves presentations of recent papers by students on algorithms and applications of deep generative models for image, video, audio and text generation and understanding as well as applications to robotics and drug/protein discovery.

• Image and Video Generation, Editing, Inpainting and Super-resolution

• Large Language Models (LLM) and Large Vision-Language Models (LVLMs)

• Few-shot, Zero-shot and Weakly-Supervised Learning using Deep Generative Models

• Deep Generative Models for Out-of-Distribution, Anomaly and Attack Detection

• Imitation Learning using Deep Generative Models

• Drug and Protein Discovery/Design using Deep Generative Models

• Other Applications of Deep Generative Models in Computer Vision

Paper summaries are due 9am on the day of the class.

• Paper Presentations (30%)

• Research Paper Synopses (25%)

• Project (40%)

• Class Participation (5%)

Project can be done individually or in teams of two people. The project typically involve leveraging existing generative models to solve an interesting application problem or exploring a new challenging dataset or extending a deep generative model learning/algorithm to address a new/existing application.

All students in the course are subject to the Northeastern University's Academic Integrity Policy. Any submitted report/homework/project by a student in this course for academic credit should be the student's own work. Collaborations are only allowed if explicitly permitted. Per CCIS policy, violations of the rules, including cheating, fabrication and plagiarism, will be reported to the Office of Student Conduct and Conflict Resolution (OSCCR). This may result in deferred suspension, suspension, or expulsion from the university.