M2 [AI] - Advanced Optimization and Automated Machine Learning

The contents of the class is accessible to all without registration under a license [CC-BY-SA]. To access quizzes and get graded you must register to the class.

Classes are held in presence in class PUIO E105 unless otherwise announced. See CALENDAR. Time is 13h30 - 16h30.

1. JAN 12: Introduction, taxonomy of optimization problems, examples of optimization problems, basic methods (stochastic gradient, grid search, random search). Solal Nathan lecturer.
   
2. JAN 19: Unconstrained continuous optimization, gradient methods, acceleration and reduction of variance of the stochastic gradient (momentum, Nesterov, Adam), true gradient methods (LBFGS, conjugate gradients). Solal Nathan lecturer.
3. JAN 26: Constrained optimization (linear programming, quadratic programming, convex programming). Solal Nathan lecturer.
4. FEB 2: Black box optimization (optimization without gradient, simulated annealing and MCMC, evolutionary methods) - Marc Schoenauer lecturer.
5. FEB 9: Hyper-parameter optimization (application of black box methods and Bayesian optimization to hyper-parameter search) - Lisheng Sun lecturer.
6. FEB16: Learning to optimize (previously we were looking at how optimization can be applied to ML, let's see now how ML can be applied to optimization) - Lisheng Sun and Hung Nguyen lecturers.
   
7. FEB 23: Neural Architecture Search (search space, constructive methods, elimination methods) - Romain Egele lecturer.
   

Continuous progress control

HOMEWORK (70%): Homework in the form of a quiz is due before class every week (see class tabs on the left side). There are 7 quizzes, each worth 10 points.

PROJECT (30%): As a mini project, you must also read and present a paper in class.

Slides, lecture notes, and recordings

The slides of the next lecture will be posted under the tab of the corresponding class (in the left menu). The class will be interactive, so presence in class is highly recommended. The classes will NOT be recorded, but the recordings from last year are available in each section.

This is a new class. The lecture notes are under construction.  Students finding mistakes or making constructive suggestions will be acknowledged!

Homework

Homework in the form of a quiz is due BEFORE CLASS every week (see class tabs on the left side). 

Office hours

Optional office hours will be held ON-LINE if the the students request it (use Collaborate link at bottom of page), to help students do their homework.

Project

As a mini project, you must also read and present a paper in class (between FEB 02 and FEB 23). Your in-class presentation will be graded. FOLLOW THIS TEMPLATE to prepare your slides.

Help

Questions should preferably be asked during class or office hours (raise your hand) or via the forum. If desperate, write to optim@chalearn.org.

Class rules

• FIRM DEADLINES. All assignment deadlines are FIRM. Both quizzes and paper reading assignments are available ahead of time and multiple submissions are permitted until the deadline. No extension will be granted.
• PLAGIARISM IS CONSIDERED CHEATING. For the purpose of the class, we will define plagiarism as the use of someone else's work (text, figures, results, code) without CITING the source. Citing does not cost you anything!! Add REFERENCES, CITE YOUR SOURCES. Students convicted of plagiarism in their slides or reading report will expose their group to having zero to the assignment. Being eligible for a second session in case of plagiarism will depend on the overall attitude of the student (see SECOND SESSION).
• TEAM WORK. Students not contributing significantly to the project will have to take a second session exam (which will consist in an individual project of an equivalent amount of work). Contributing includes coming to class for the presentation, and answering the questions of the other students and the instructor.
• GETTING HELP. There is a quiz every week that must be completed (individually) before class. You can ask for help, but each person must turn in their own quiz. Asking for help or looking up the answer (on the Internet or elsewhere) is NOT considered cheating. 
• PARTICIPATION. We welcome questions or comments during class (raise your hand). There are no dumb questions.
• SECOND SESSION. The grades of the quizzes cannot be upgraded. A written reading report can be turned in to improve the grade, in lieu of second session exam. The permission to enter the second session, exact modalities and deadline will be announced after the first session jury deliberation and will depend (among other things) on the date of the second session jury. The attitude of the student, including turning in all assignments on time, being present in class and at the office hours, and asking questions, will be taken into account to determine eligibility  for the second session. 

Class schedule

Except for the first 2 classes (lectures only) the schedule will be as follows: