Introduction to Online Optimization/Learning (Fall 2022)

Project Overview

The final project is to conduct a short survey on a specific topic related to online learning. Each of you has been assigned one paper, based on alphabetical order of last names (unless you requested a specific topic). The project includes two parts (more instructions on both parts can be found after the schedule table):

(30%) an in-class 15-min presentation on the assigned paper, on the assigned date.
(30%) a final survey report, due on 12/12 (submit through Blackboard; no late day is allowed).

Assigned Papers and Schedule

Date	Papers	Presenters
11/10	Logarithmic Regret Algorithms for Online Convex Optimization	Mohammadamin Banayeeanzade
	Adaptive Subgradient Methods for Online Learning and Stochastic Optimization	Rashmi Ranjan Bhuyan
	On The Convergence of ADAM and Beyond	Robby Costales
	Online Non-Convex Learning: Following the Perturbed Leader Is Optimal	Wei Gu
	Anytime Online-to-Batch Conversions, Optimism, and Acceleration	James Enouen
	Online Learning with Imperfect Hints	Wang Zhu
	Parameter-free, Dynamic, and Strongly-Adaptive Online Learning	Samuel Griesemer
	The Multiplicative Weights Update Method: a Meta Algorithm and Applications	Yihang Zhang
	Soft-Bayes: Prod for Mixtures of Experts with Log-Loss	Jingwei Ji
	Advancing Subgroup Fairness via Sleeping Experts	Siddartha Devic
11/17	Efficient Online Learning for Dynamic k-Clustering	Miryam Huang
	Learning in Games: Robustness of Fast Convergence	Zizhao Hu
	From External to Internal Regret	Amin Jabini
	Faster Rates for Convex-Concave Games	Aravinda Kolla
	An Information-Theoretic Analysis of Thompson Sampling	Dongze Ye
	Knapsack Based Optimal Policies for Budget–Limited Multi–Armed Bandits	Yan Wen
	Bandits with Switching Costs: T^{2/3} Regret	Xinyu Mao
	Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problems	Ram Deo-Campo Vuong
	Bandit Samplers for Training Graph Neural Networks	Ta-Yang Wang
12/01	The End of Optimism? An Asymptotic Analysis of Finite-Armed Linear Bandits	Neel Patel
	Adapting to Misspecification in Contextual Bandits	Xiao Song
	Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discrimination	Zhuozheng Sun
	Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizability	Jeel Tejaskumar Vaishnav
	Bandit Convex Optimization: Towards Tight Bounds	Mengxi Wu
	Bandit Convex Optimization: √T Regret in One Dimension	Changzhi Xie
	Is Q-learning Provably Efficient?	Yinbin Han
	Provably Efficient Reinforcement Learning with Linear Function Approximation	Deqing Fu
	On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift	Akhil Agnihotri

Guidance on Reading the Papers

Note that these papers are all closely related to the materials covered in the lectures. So when reading them, try to find the connections with what you have learned from the lectures.
Some papers could be notation-heavy and get rather technical at some parts. You are by no mean required to understand every single detail of the paper. Instead, try to focus on the key ideas/messages. It is fine that you only skim some parts of the paper, as long as you spend time in carefully reading and understanding some other main parts.
If you need further help understanding the papers, feel free to ask on Piazza or come to the office hours.

Instructions on the Presentation

The goal of this part is to give you an opportunity to practice the skill of giving talks, which is an extremely important part of your PhD training (but is often overlooked). Follow the instructions below to make sure you give a smooth talk.

Each presentation is 15 minute long plus 5 minutes of Q&A and transition. It is recommended that you roughly break it down into following parts:

(1-2 minutes) introduction: what is the topic you are presenting? how is this related to the lectures? what are the main results of the assigned paper in a few words? why are they significant? what will you cover in the rest of the talk?
(2-4 minutes) setting up: describe the problem and introduce necessary notations.
(8-10 minutes) main results: describe the main algorithms and/or theorems, with emphasis on explaining the key ideas.
(1-2 minutes) conclusion: conclude you talk and highlight the key take-home message.

In addition, keep the following in mind when preparing the talks:

You can either give a board talk (just like our lectures) or use slides for the presentation. For a board talk, think carefully about how you are going to use the board: how much space do I need? what should I put in each part of the board? how do I avoid erasing stuffs that I need later? If you are using slides instead, please let me know before the lecture so we can make sure there is no problem in setting up the projector.

Anticipate questions from the audience. Just like the lectures, everyone is encouraged to ask questions during the talk. Make sure you understand the question before answering, and try to answer it concisely so you can finish the talk in time. It is perfectly fine if you do not know the answer for tough questions — simply acknowledge it in this case and move on. But make sure you address simple clarification questions, especially at the beginning of the talk when setting up the problems.

By no mean you need to cover everything in your assigned paper! In fact, most likely you will only have time to cover one or two main results. Keep this in mind and make the key message of your talk clear.

Even if the paper is notation-heavy, it does not mean your talk has to be so. Try to use a minimum amount of notation to convey what you want to say. It is common to make simplification in talks even if that means sacrificing generality or even rigorousness.

If you have any other questions, feel free to ask on Piazza!

Instructions on Writing the Report

The final report has to be written in Latex and must be at least 6-page long, excluding references. You can use the NeurIPS format that we have been using for the homework, or some other similar styles. The report summarizes and distills the main results of your assigned paper, and should include the following components:

A short abstract, summarizing the entire survey.
Introduction: Introduce the main topic of you assigned paper. Try to put it into the context of online learning literature and explain how it relates to topics covered in our lectures. Briefly mention the high-level results and explain the significance of the results (such as improvement over prior work).
Problem setup: For this and the next two parts (“Main results” and “Proofs”), you can use the lecture notes as examples. For problem setup, you should describe the problems in detail using necessary notation. Once again, you are not required to cover everything in the paper, so only describe in detail what you plan to cover in this short survey.
Main results: Describe the main algorithms/theorems. For algorithms, describe what they are doing at each step and what the key ideas are. For theorems, after the formal statement, try to explain in words what the statement really means and what the implications are.
Proofs: Try to distill some proofs from the paper and reproduce them in the report. Pick the ones that you think are the most important or inspiring to you. If the original proofs are long and complicated, try to break it down into several parts (in the form of lemmas for example), and only present the proofs for some of these parts. Again, use the lecture notes as examples.
A short conclusion, highlighting the main message again.
References.

The following two are optional and considered as bonus tasks.

Open questions: Identify interesting and concrete open questions in the same direction that are not mentioned in the papers already. Mention briefly what you think the potential approaches are to tackle these open questions and/or why you think these are hard problems that require new techniques beyond what the paper already presents.
More papers in the same direction: Read more related papers on the same topic and summarize what they are about and how they are related/compared to your assigned paper. To find these other papers, you can use the reference list of your assigned paper, use Google Scholar to see which papers cite your assigned paper, or do a general online search on the related topic.

If you have any other questions, feel free to ask on Piazza!