CSCI 699: Robot Learning

Fall 2024-2025, Class: Fri 1:00-4:20pm, KAP 158

Syllabus | Piazza | Gradescope

Description:

Robot learning is an interdisciplinary field at the intersection of robotics, machine learning, cognitive science, and control theory, aiming to create intelligent and adaptable robotic systems capable of learning from their environment and experience. With rapid advances in artificial intelligence and computing power, as well as the possibility of having larger datasets, robot learning has the potential to revolutionize a wide range of applications, from manufacturing and healthcare to transportation and personal assistance. However, developing learning algorithms for real-world robotic systems poses unique challenges due to the complexities of the physical world, safety concerns, and the need for efficient and robust learning methods.

This course provides a comprehensive introduction to the fundamentals of robot learning, covering topics such as reinforcement learning, computer vision, meta-learning, sim-to-real transfer, and multi-agent learning. Students will explore cutting-edge techniques in imitation learning, inverse reinforcement learning, representation learning, and safe and robust learning, while also discussing the real-world applications and challenges of robot learning. The course is designed to be accessible to PhD students in robotics, control theory, machine learning, artificial intelligence, optimization, and related fields; with an emphasis on both theoretical foundations and practical applications.

In addition to lectures, the course features a series of student-led presentations on recent research papers and a course project, allowing students to gain hands-on experience with the latest advances in robot learning and explore emerging research topics. Through a combination of lectures, homework assignments, presentations, and project work, students will develop a deep understanding of robot learning techniques and their potential to transform the way we interact with and utilize robots in our everyday lives.

Prerequisites:

Students are recommended to have familiarity with fundamental concepts in machine learning. [CSCI 467: Introduction to Machine Learning AND (CSCI 445L: Introduction to Robotics OR CSCI 545: Robotics)] are recommended but not required.

Staff

Erdem Bıyık

Instructor

Office Hours: Fri 11:00AM-12:00PM

Location: PHE 214

biyik [at] usc [dot] edu

Webpage

Ishika Singh

Teaching Assistant

Office Hours: Thu 3:30PM-4:30PM

Location: RTH 4th Floor Lounge

ishikasi [at] usc [dot] edu

Webpage

Timeline

Date	Lecture	Readings / Deadlines	Notes
Week 1 Fri, Aug 30	General course information Lecture Basics of robotics Lecture Fundamentals of machine learning	Please checkout our Course Policies.	Slides
Week 2 Fri, Sep 6	Lecture Basics of computer vision for robotics Lecture Representation learning		Homework 1 Slides
Week 3 Fri, Sep 13	Presentation Representation learning Lecture Reinforcement learning	Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020). Zhou et al., Unsupervised Learning of Depth and Ego-Motion from Video (2017). Oord et al., Representation Learning with Contrastive Predictive Coding (2018). Chen et al., A Simple Framework for Contrastive Learning of Visual Representations (2020). Bobu et al., SIRL: Similarity-based Implicit Representation Learning (2023). Liang et al., ViSaRL: Visual Reinforcement Learning Guided by Human Saliency (2024).	Slides
Week 4 Fri, Sep 20	Lecture Reinforcement learning Presentation Reinforcement learning	Due Homework #1 Janner et al., When to Trust Your Model: Model-Based Policy Optimization (2019). Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination (2019). Srinivas et al., CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020). Sontakke et al., RoboCLIP: One Demonstration is Enough to Learn Robot Policies (2023). Ajay et al., Is Conditional Generative Modeling All You Need for Decision-making? (2023). Goyal et al., RVT: Robotic View Transformer for 3D Object Manipulation (2023).	Slides
Week 5 Fri, Sep 27	Presentation Reinforcement learning Lecture Imitation learning	Haarnoja et al., Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018). Schulman et al., Proximal Policy Optimization Algorithms (2017). Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling (2021). Reddy et al., Shared Autonomy via Deep Reinforcement Learning (2018). Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023). Goyal et al., RVT-2: Learning Precise Manipulation from Few Demonstrations (2024).	Homework 2 Slides
Week 6 Fri, Oct 4	Presentation Imitation learning Lecture Learning from human feedback	Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2011). Ho and Ermon, Generative Adversarial Imitation Learning (2016). Florence et al., Implicit Behavioral Cloning (2021). Shafiullah et al., Behavior Transformers: Cloning k modes with one stone (2022). Jain et al., Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers (2024). Fu et al., In-Context Imitation Learning via Next-Token Prediction (2024).	Slides
Week 7 Fri, Oct 11	Fall Recess: No Lecture		Homework 3
Week 8 Fri, Oct 18	Lecture Learning from human feedback Presentation Learning from human feedback	Due Homework #2 Due Project Proposal Myers et al., Active Reward Learning from Online Preferences (2023). Bobu et al., Inducing Structure in Reward Learning by Learning Features (2022). Hadfield-Menell et al., Inverse Reward Design (2017). Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2024). Kwon et al., When Humans aren't Optimal: Robots that Collaborate with Risk-aware Humans (2020). Awad et al., The Moral Machine Experiment (2018).
Week 9 Fri, Oct 25	Lecture Sim-to-real transfer Presentation Sim-to-real transfer	Bajcsy et al., Learning Robot Objectives from Physical Human Interaction (2017). Peng et al., Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017). Matas et al., Sim-to-Real Reinforcement Learning for Deformable Object Manipulation (2018). James et al., Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks (2019). Du et al., Auto-Tuned Sim-to-Real Transfer (2021). Nasiriany et al., RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024).	Slides
Week 10 Fri, Nov 1	Lecture Meta & Multi-task learning Presentation Meta & Multi-task learning	Due Homework #3 Chan et al., Human Irrationality: Both Bad and Good for Reward Inference (2021). Julian et al., Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning (2020). Kim et al., Bayesian Model-Agnostic Meta-Learning (2018). Zintgraf et al., VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2020). Sodhani et al., Multi-Task Reinforcement Learning with Context-based Representations (2021). Shridhar et al., Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022).	Slides
Week 11 Fri, Nov 8	Guest Lecture Prof. Heather Culbertson Guest Lecture Dr. Aaquib Tabrez	Understanding Material Properties Through Haptic Data Multimodal Explanation-based Reward Coaching and Decision Support to Improve Human-Robot Teaming
Week 12 Fri, Nov 15	Presentation Safe and robust learning Lecture Multi-agent learning	Due Project Milestone Report Sui et al., Safe Exploration for Optimization with Gaussian Processes (2015). Achiam et al., Constrained Policy Optimization (2017). Robey et al., Learning Control Barrier Functions from Expert Demonstrations (2020). Bansal and Tomlin, Deepreach: A Deep Learning Approach to High-dimensional Reachability (2021). Hsu et al., The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems (2024). Huang et al., ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024).	Slides
Week 13 Fri, Nov 22	Presentation Multi-agent learning Presentation Robot learning using natural language	Jeon et al., Shared Autonomy with Learned Latent Actions (2020). Foerster et al., Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016). Silver et al., Mastering the Game of Go with Deep Neural Networks and Tree Search (2016). Hu et al., “Other-Play” for Zero-Shot Coordination (2021). Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022). Sharma et al., Correcting Robot Plans with Natural Language Feedback (2022). Singh et al., ProgPrompt: Generating Situated Robot Task Plans using Large Language Models (2023). Brohan et al., RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023). Gervet et al., Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation (2023). Ke et al., 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024). Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model (2024). Duan et al., Manipulate-Anything: Automating Real-World Robots using Vision-Language Models (2024).
Week 14 Fri, Nov 29	Thanksgiving Break: No Lecture
Week 15 Fri, Dec 6	Project Project Presentations	Due Final Project Report
Finals Week Fri, Dec 13	No lecture. No final exam.	Due Peer Review

Grading Metrics

Component	Contribution to Grade
Homework	45%
Class Presentations	15%
Course Project	40%
Total	100%

Project Grading

Component	Contribution to Grade
Project Proposal Report	5%
Project Milestone Report	5%
Project Presentation (Possibly with Demo)	10%
Final Project Report	15%
Peer Review	5%
Total	40%

Grading Policies

Homework (15%): Students will be assigned three homework sets that consist of both report questions and programming questions (in Python). Each student will select one of the three homework sets to complete. Report questions will require students to work on problems related to past lectures with pen and paper. Programming questions will require students to implement some of the methods covered in the lectures, occasionally with further improvements, and experiment them on simulated robot environments and/or machine learning tasks.

Class Presentation (45%): Students will present three research papers from literature. Presentations will be followed by open discussions. Students will be graded based on their presentations.

Course Project (40%): Students will be required to work on a course project in groups of 2-3. The projects must have both robotics and machine learning components. They can be, for example, application-dependent improvements over an existing robot learning method, a novel robot learning related application of an existing technique, or a completely new method that may have potential benefits. Students will write a 2-page project proposal, present their findings in an oral presentation, write a conference paper-style 6-8 pages project report, and write an anonymous peer review (max 1 page) for the project report of another group. There will be a 2-page project milestone along the way to guide progress. Instructor and teaching assistant(s) will provide feedback on the project milestone.

Other References

This class is partially based on the following existing courses: