CS 5955/6955 Advanced Artificial Intelligence

Class Overview

This course focuses on advanced algorithms for intelligent sequential decision making with a focus on modern deep learning-based methods. The class will cover both the theory and practical details of the algorithms behind recent breakthroughs in many types of AI decision making, including game playing, robotics, recommendation systems, and large language models. Topics include bandit algorithms, Markov decision processes, partially observable Markov decision processes, reinforcement learning, imitation learning, inverse reinforcement learning, and reinforcement learning from human feedback. This will be a fun, but challenging class. It is an advanced AI class so we will assume a basic understanding of machine learning basics (supervised learning, loss functions, gradient descent) and a basic understanding of AI basics (search problems, MDPs, RL high-level ideas). Note that these topics can be picked up during the class as we will try to keep things self-contained, but we will go over basic topics quickly to get to more advanced materials. Students should be comfortable writing Python code and digging through and understanding code written by others.

Class Schedule

Date Topic Slides Readings Assignment
Jan 6 Class Intro Slides Optional: Python Notes (Alan Kuntz) Optional: Python Tutorial (Berkeley AI Class)
Jan 8 Behavioral Cloning Slides Optional: Behavioral Cloning from Observation, DAgger, ThriftyDAgger Behavior Cloning in PyTorch (due Friday Jan 17)
Jan 13 Intro to Advanced Behavior Cloning Slides Choose one and submit reading report before class: Implicit Behavioral Cloning, Action Chunking Transformer, Diffusion Policy
Jan 15 More Advanced Behavior Cloning Slides
Jan 22 Multi-Armed Bandits and Evaluative Feedback Slides Sutton and Barto 2.1-2.5 Multi-Armed Bandits (due Fri Jan 31)
Jan 27 More Bandits Slides
Jan 29 Intro to Markov Decision Processes Slides
Feb 3 Solving MDPs
Feb 5 Value-Based RL
Feb 10 Policy-Based RL
Feb 12 AlphaGo and AlphaZero
Feb 19 Advanced Deep RL
Feb 24 More Advanced Deep RL
Feb 26 Multi-Agent RL
Mar 3 RL from Human Feedback (RLHF)
Mar 5 More RL from Human Feedback (RLHF)

Additional Resources

Here you can find supplementary materials, links, etc.

PyTorch Tutorials