CSCI 642: Robot Learning

Fall 2026-2027, Class: Wed 3:30-6:50pm, DMC 100

Syllabus | Piazza | Gradescope

Description:

Robot learning is an interdisciplinary field at the intersection of robotics, machine learning, cognitive science, and control theory, aiming to create intelligent and adaptable robotic systems capable of learning from their environment and experience. With rapid advances in artificial intelligence and computing power, as well as the possibility of having larger datasets, robot learning has the potential to revolutionize a wide range of applications, from manufacturing and healthcare to transportation and personal assistance. However, developing learning algorithms for real-world robotic systems poses unique challenges due to the complexities of the physical world, safety concerns, and the need for efficient and robust learning methods.

This course provides a comprehensive introduction to the fundamentals of robot learning, covering topics such as reinforcement learning, computer vision, meta-learning, sim-to-real transfer, and multi-agent learning. Students will explore cutting-edge techniques in imitation learning, inverse reinforcement learning, representation learning, and safe and robust learning, while also discussing the real-world applications and challenges of robot learning. The course is designed to be accessible to PhD students in robotics, control theory, machine learning, artificial intelligence, optimization, and related fields; with an emphasis on both theoretical foundations and practical applications.

In addition to lectures, the course features a series of student-led presentations on recent research papers and a course project, allowing students to gain hands-on experience with the latest advances in robot learning and explore emerging research topics. Through a combination of lectures, homework assignments, presentations, and project work, students will develop a deep understanding of robot learning techniques and their potential to transform the way we interact with and utilize robots in our everyday lives.

Prerequisites:

Students are recommended to have familiarity with fundamental concepts in machine learning. [CSCI 467: Introduction to Machine Learning AND (CSCI 445L: Introduction to Robotics OR CSCI 545: Robotics)] are recommended but not required.

Staff

Erdem Bıyık

Instructor

Office Hours: Wed 2:00PM-3:00PM

Location: GCS SB3 (Floor: LL2)

biyik [at] usc [dot] edu

Webpage

Timeline

Date	Lecture	Readings / Deadlines	Notes
Week 1 Wed, Aug 26	General course information Lecture Basics of robotics Lecture Fundamentals of machine learning	Please checkout our Course Policies. Zhou et al., Unsupervised Learning of Depth and Ego-Motion from Video (2017). Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020).	Slides
Week 2 Wed, Sep 2	Lecture Basics of computer vision for robotics Lecture Representation learning	Chen et al., A Simple Framework for Contrastive Learning of Visual Representations (2020). Bobu et al., SIRL: Similarity-based Implicit Representation Learning (2023).	Homework 1 Slides
Week 3 Wed, Sep 9	Lecture Dynamic programming Lecture Tabular reinforcement learning	Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination (2019). Haarnoja et al., Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018).
Week 4 Wed, Sep 16	Lecture Model-based reinforcement learning Lecture Model-free reinforcement learning	Due Homework #1 Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling (2021). Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (2023).
Week 5 Wed, Sep 23	Lecture Imitation learning	Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning (2011). Ho and Ermon, Generative Adversarial Imitation Learning (2016).	Homework 2
Week 6 Wed, Sep 30	Lecture Inverse reinforcement learning	Florence et al., Implicit Behavioral Cloning (2021). Shafiullah et al., Behavior Transformers: Cloning k modes with one stone (2022).
Week 7 Wed, Oct 7	Lecture Learning from human feedback	Hadfield-Menell et al., Inverse Reward Design (2017). Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2024).	Homework 3
Week 8 Wed, Oct 14	Lecture Human-in-the-loop robot learning	Due Homework #2 Due Project Proposal Kwon et al., When Humans aren't Optimal: Robots that Collaborate with Risk-aware Humans (2020). Awad et al., The Moral Machine Experiment (2018).
Week 9 Wed, Oct 21	Lecture Sim-to-real transfer Exam Midterm	Bajcsy et al., Learning Robot Objectives from Physical Human Interaction (2017). Jeon et al., Shared Autonomy with Learned Latent Actions (2020).
Week 10 Wed, Oct 28	Lecture Meta learning Lecture Multi-task learning	Due Homework #3 Peng et al., Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017). James et al., Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks (2019).
Week 11 Wed, Nov 4	Lecture Multi-agent learning	Due Project Milestone Report Zintgraf et al., VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning (2020). Shridhar et al., Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation (2022).
Week 12 Wed, Nov 11	Veterans Day Holiday: No Lecture	Achiam et al., Constrained Policy Optimization (2017). Hsu et al., The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems (2024).
Week 13 Wed, Nov 18	Lecture Robot learning using natural language	Silver et al., Mastering the game of Go with deep neural networks and tree search (2016). Foerster et al., Learning to communicate with deep multi-agent reinforcement learning (2016).
Week 14 Wed, Nov 25	Thanksgiving Break: No Lecture	Ahn et al., Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022). Brohan et al., RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023).
Week 15 Wed, Dec 2	Project Project Presentations	Due Final Project Report
Finals Week Wed, Dec 9	No lecture. No final exam.	Due Peer Review

Grading Metrics

Component	Contribution to Grade
Readings	7%
Homework	20%
Class Presentations	15%
Midterm Exam	15%
Course Project	38%
Peer Review	5%
Total	100%

Project Grading

Component	Contribution to Grade
Project Proposal Report	5%
Project Milestone Report	8%
Project Presentation (Possibly with Demo)	10%
Final Project Report	15%
Total	38%

Grading Policies

Readings (7%): Students will write summaries of the required readings every week. For each reading, the summary is expected to include 1-2 sentence description of the problem the paper is trying to solve, 2-3 sentences of the core novel ideas or technical contributions of the paper, 3-4 sentences of what specific techniques it uses, and 1-2 sentences of its main findings.

Homework (20%): Students will be assigned three homework sets that consist of both report questions and programming questions (in Python). Each student will select two of the three homework sets to complete. Report questions will require students to work on problems related to past lectures with pen and paper. Programming questions will require students to implement some of the methods covered in the lectures, occasionally with further improvements, and experiment them on simulated robot environments and/or machine learning tasks.

Class Presentation (15%): Students will present 1 research paper from the presentation papers list in the syllabus. In the beginning of the semester, a sign-up sheet will be released for students to claim papers for presentations. After all students sign up, the presentations will be scheduled by the instructor to ensure presentations take place in the classes that are on the relevant topic. Presentations will be followed by open discussions. Students will be graded based on their presentations.

Midterm Exam (15%): One midterm exam will be taken during the scheduled class time. The students will individually answer the exam questions by pen/pencil on paper, without access to books or electronic devices. The teaching staff will make every effort to return graded exams within one week. Students will then have one week from the return date to raise any grading-related issues.

Course Project (38%): In addition, students will be required to work on a course project in groups of 2-3. The projects must have both robotics and machine learning components. They can be, for example, application-dependent improvements over an existing robot learning method, a novel robot learning related application of an existing technique, or a completely new method that may have potential benefits. Students will write a project proposal (2 pages excluding references), write a milestone report (2 pages excluding references), present their findings in an oral presentation (15 minutes excluding Q&A), and write a conference paper-style project report (6-8 pages excluding references). Instructor and teaching assistant(s) will provide feedback to the students on their proposal and milestone reports to guide progress.

Peer Review (5%): Students will be randomly assigned a final project report of another group. Each student will individually write a one-page peer review for that report. The students will be assessed based on the quality of the peer review they write. The reviews will be shared with the group members, but they will not affect the grade of the group.

Other References

This class is partially based on the following existing courses: