CSCI 699: Robot Learning

Fall 2023-2024, Class: Fri 1:00-4:20pm, THH 208

Syllabus | Piazza | Gradescope

Description:

Robot learning is an interdisciplinary field at the intersection of robotics, machine learning, cognitive science, and control theory, aiming to create intelligent and adaptable robotic systems capable of learning from their environment and experience. With rapid advances in artificial intelligence and computing power, as well as the possibility of having larger datasets, robot learning has the potential to revolutionize a wide range of applications, from manufacturing and healthcare to transportation and personal assistance. However, developing learning algorithms for real-world robotic systems poses unique challenges due to the complexities of the physical world, safety concerns, and the need for efficient and robust learning methods.

This course provides a comprehensive introduction to the fundamentals of robot learning, covering topics such as reinforcement learning, computer vision, meta-learning, sim-to-real transfer, and multi-agent learning. Students will explore cutting-edge techniques in imitation learning, inverse reinforcement learning, representation learning, and safe and robust learning, while also discussing the real-world applications and challenges of robot learning. The course is designed to be accessible to PhD students in robotics, control theory, machine learning, artificial intelligence, optimization, and related fields; with an emphasis on both theoretical foundations and practical applications.

In addition to lectures, the course features a series of student-led presentations on recent research papers and a course project, allowing students to gain hands-on experience with the latest advances in robot learning and explore emerging research topics. Through a combination of lectures, homework assignments, presentations, and project work, students will develop a deep understanding of robot learning techniques and their potential to transform the way we interact with and utilize robots in our everyday lives.

Prerequisites:

Students are recommended to have familiarity with fundamental concepts in machine learning. CSCI 467: Introduction to Machine Learning, and CSCI 445L: Introduction to Robotics are recommended but not required.

Staff

Erdem Bıyık

Instructor

Office Hours: Fri 11AM-12PM

Location: PHE 214

biyik [at] usc [dot] edu

Webpage

Anthony Liang

Teaching Assistant

Office Hours: Thu 1PM-2PM

Location: RTH Floor 4 Lounge

aliang80 [at] usc [dot] edu

Webpage

Pragya Goel

Grader

p872038 [at] usc [dot] edu

Timeline

Date	Lecture	Readings / Deadlines	Notes
Week 1 Fri, Aug 25	General course information Lecture Basics of robotics Lecture Fundamentals of machine learning	Please checkout our Course Policies.	Slides
Week 2 Fri, Sep 1	Lecture Basics of computer vision for robotics Lecture Representation learning		Homework 1 Slides
Week 3 Fri, Sep 8	Presentation Representation learning Lecture Reinforcement learning	Zhou et al., Unsupervised Learning of Depth and Ego-Motion from Video (2017). Oord et al., Representation Learning with Contrastive Predictive Coding (2018). Chen et al., A Simple Framework for Contrastive Learning of Visual Representations (2020). Bobu et al., SIRL: Similarity-based Implicit Representation Learning (2023).	Slides
Week 4 Fri, Sep 15	Lecture Reinforcement learning Presentation Reinforcement learning	Due Homework #1 Janner et al., When to Trust Your Model: Model-Based Policy Optimization (2019). Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination (2019). Srinivas et al., CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020). Ajay et al., Is Conditional Generative Modeling All You Need for Decision-making?, (2023).	Slides
Week 5 Fri, Sep 22	Presentation Reinforcement learning Lecture Imitation learning	Haarnoja et al., Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018). Schulman et al., Proximal Policy Optimization Algorithms (2017). Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling (2021). Reddy et al., Shared Autonomy via Deep Reinforcement Learning (2018).	Homework 2 Slides
Week 6 Fri, Sep 29	Presentation Imitation learning Lecture Learning from human feedback	Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2011). Ho and Ermon, Generative Adversarial Imitation Learning (2016). Florence et al., Implicit Behavioral Cloning (2021). Shafiullah et al., Behavior Transformers: Cloning k modes with one stone (2022).	Slides
Week 7 Fri, Oct 6	Lecture Learning from human feedback Presentation Learning from human feedback	Due Homework #2 Due Project Proposal Myers et al., Active Reward Learning from Online Preferences (2023). Bobu et al., Inducing Structure in Reward Learning by Learning Features (2022). Hadfield-Menell et al., Inverse Reward Design (2017). Kwon et al., When Humans aren't Optimal: Robots that Collaborate with Risk-aware Humans (2020).
Week 8 Fri, Oct 13	Fall Recess: No Lecture		Homework 3
Week 9 Fri, Oct 20	Lecture Sim-to-real transfer Presentation Sim-to-real transfer	Bajcsy et al., Learning Robot Objectives from Physical Human Interaction (2017). Peng et al., Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017). Matas et al., Sim-to-Real Reinforcement Learning for Deformable Object Manipulation (2018). James et al., Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks (2019). Du et al., Auto-Tuned Sim-to-Real Transfer (2021).	Slides
Week 10 Fri, Oct 27	Lecture Meta & Multi-task learning Presentation Meta & Multi-task learning	Due Homework #3 Chan et al., Human Irrationality: Both Bad and Good for Reward Inference (2021). Julian et al., Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning (2020). Kim et al., Bayesian Model-Agnostic Meta-Learning (2018). Sodhani et al., Multi-Task Reinforcement Learning with Context-based Representations (2021). Shridhar et al., Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022).	Slides
Week 11 Fri, Nov 3	Presentation Safe and robust learning Lecture Multi-agent learning	Due Project Milestone Report Sui et al., Safe Exploration for Optimization with Gaussian Processes (2015). Achiam et al., Constrained Policy Optimization (2017). Robey et al., Learning Control Barrier Functions from Expert Demonstrations (2020). Bansal and Tomlin, Deepreach: A Deep Learning Approach to High-dimensional Reachability (2021).	Slides
Week 12 Fri, Nov 10	Veterans Day: No Lecture
Week 13 Fri, Nov 17	Presentation Multi-agent learning Presentation Robot learning using natural language	Jeon et al., Shared Autonomy with Learned Latent Actions (2020). Foerster et al., Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016). Silver et al., Mastering the Game of Go with Deep Neural Networks and Tree Search (2016). Hu et al., “Other-Play” for Zero-Shot Coordination (2021). Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022). Sharma et al., Correcting Robot Plans with Natural Language Feedback (2022). Singh et al., ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2023). Brohan et al., RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023).
Week 14 Fri, Nov 24	Thanksgiving Break: No Lecture
Week 15 Fri, Dec 1	Project Project Presentations	Due Final Project Report
Finals Week Fri, Dec 8	No lecture. No final exam.	Due Peer Review

Grading Metrics

Component	Contribution to Grade
Homework	45%
Class Presentations	15%
Course Project	40%
Total	100%

Project Grading

Component	Contribution to Grade
Project Proposal Report	5%
Project Milestone Report	5%
Project Presentation (Possibly with Demo)	10%
Final Project Report	15%
Peer Review	5%
Total	40%

Grading Policies

Homework (45%): Students will be assigned three homework sets that consist of both report questions and programming questions (in Python). Report questions will require students to work on problems related to past lectures with pen and paper. Programming questions will require students to implement some of the methods covered in the lectures, occasionally with further improvements, and experiment them on simulated robot environments and/or machine learning tasks.

Homework reports and codes will be submitted online. Students will have a total of 8 free late days that may be used for the homework assignments; a maximum of 4 late days will be allowed on a given assignment. Late days are only for homework assignments, and cannot be used for the class presentation or deadlines related to the course project.

Class Presentation (15%): Students will present research papers from literature. Presentations will be followed by open discussions. Students will be graded based on their presentations.

In case the class size is too large to have every student make a presentation, some students will be required to write a short review of the papers that will be presented in class. These short reviews will be due on the beginning of the class.

Course Project (40%): Students will be required to work on a course project in groups of 2-3. The projects must have both robotics and machine learning components. They can be, for example, application-dependent improvements over an existing robot learning method, a novel robot learning related application of an existing technique, or a completely new method that may have potential benefits. Students will write a 2-page project proposal, present their findings in an oral presentation, write a conference paper-style 6-8 pages project report, and write an anonymous peer review (max 1 page) for the project report of another group. There will be a 2-page project milestone along the way to guide progress. Instructor and teaching assistant(s) will provide feedback on the project milestone.

Other References

This class is partially based on the following existing courses: