# AutoML (Humania) reading group

Organizers: Haozhe (sunhaozhe275940200@gmail.com), Heri (heri@lri.fr), Zhengying (zhengying.liu@inria.fr)

Link to Zotero AutoML group

Even though this Reading Group is called AutoML due to historical reason, you are free to present papers that you wish to share in your own field or elsewhere.

**Next meeting**

**TBA**

Paper: TBA

Presenter: TBA

**Cycle of presenters**

Haozhe, Adrian, Adrien, Herilalaina, Romain, Saumya, Zhen, Zhengying

If you are not in this list but want to give presentations, please send a message to the organizers.

**Practical information**

**During the Covid period, the meetings will be online via Skype. The**** link is here: **

Legacy: Room 2014, DIGITEO (Bat. 660) or via **video call ****link**

Add to calendar / TAU calendar (Zimbra login required)

Place to put questions during the presentation: Overleaf link

**To-Read List**

Link to this list in Zotero group.

Everyone is welcome to suggest and add papers to read (send your Zotero username or email address to the organizers).

**Participants**

"tau-seminars" <tau-seminars@inria.fr> (only send to this mail)

"tau" <tau@lri.fr>

"guillaume charpiat" <guillaume.charpiat@inria.fr>

"Isabelle Guyon" <guyon@clopinet.com>

"Michele Sebag" <Michele.Sebag@lri.fr>

"doquet" <Guillaume.Doquet@lri.fr>

"fissore" <giancarlo.fissore@lri.fr>

"Marc Schoenauer" <marc.schoenauer@inria.fr>

"Laurent Cetinsoy" <laurent.cetinsoy@gmail.com>

"loris felardos" <loris.felardos@lri.fr>

Michael Vaccaro <michael.vaccaro@student-cs.fr>

Hugo Sonnery <hugo.sonnery@student-cs.fr>

**Past Meetings**

**April 21****, 2021 **

Paper: Hyperparameter Ensembles for Robustness and Uncertainty Quantification

Presenter: Romain

Participants: Haozhe, Zhengying, Marc, Zak

Takeaway:

building an ensemble is easy

an ensemble of models can be used for uncertainty quantification

diversity is important in an ensemble

**April 14****, 2021 **

Paper: Meta Pseudo Labels

Presenter: Zhengying

Participants: Adrien, Adrian, Zhen, Haozhe, Alessandro, Hung Nguyen

Takeaway:

Use a (constantly changing) teacher network and student network to achieve semi-supervised learning

Interesting ideas for semi-supervised learning, integrating ideas from meta-learning, NAS and other fields

Achieved 90.2% top-1 accuracy on ImageNet

**March 17, 2021 **

Paper: Free Lunch for Few-shot Learning: Distribution Calibration

Presenter: Heri

Participants: Adrien, Adrian, Zhengying

Takeaway: One can bypass expensive fine tuning with a simple distribution calibration over the embedded features. This propose to estimate data statistics (mean and variance) of target task by transferring statistics of neighbor classes used in Train. Then, target data statistics are used to generate more examples on the support set. And finally, they train simple classifier over the augmented data. Their approach beats some complex and expensive few shot learners.

**March 10, 2021 **

Paper: Cross-validation for selecting a model selection procedure

Presenter: Adrien

Participants: Adrian, Haozhe, Zhen, Zhengying

**Takeaway**: the following 3 statements about cross-validation (CV) are **NOT generally true**:

Leave-one-out (LOO) CV has smaller bias but larger variance than K-Fold CV

Better estimation of the prediction error by CV means better model selection

The best method to use for model selection is 10-fold CV

*(but actually it seems to perform well and is a good trade-off)*

**February 10, 2021 **

Paper: Compositional generalization through meta sequence-to-sequence learning

Presenter: Adrian

Participants: Zhen, Haozhe, Isabelle, Michael, Adrien

Takeaway: compositional learning on SCAN tasks (seq2seq) created some ground to experiment the selection of meta-training data to induce inductive biases into a model. It would be interesting to see if similar experiments can be performed on more complex data (e.g. images, ...).

**January 27, 2021 **

Paper: Meta-Learning Symmetries by Reparameterization

Presenter: Haozhe Sun

Participants: Adrian, Romain, Zhen, Zhengying

Takeaway: this paper presents a way to automatically learn equivariance from data that share certain common underlying symmetry. This approach uses matrix factorization and meta-learning framework.

**July 2, 2020 **

Paper: An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Presenter: Dhiaeddine YOUSFI

**Meeting of 11/06/2020**

Paper: The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Presenter: Zhengying

Participants: all TAUers were there!

**Main idea of the article: Empirical work that states a hypothesis on the existence of winning tickets (sub-networks that train faster with better performance) and shed on insights on the behavior of SGD. **

**Meeting of 04/06/2020**

Paper: DAG-GNN: DAG Structure Learning with Graph Neural Networks

Presenter: Shou Xiao

Participants: Heri, Isabelle, Kristin Bennett, Zhengying

**Main idea of the article: Structure Learning - learning a weighted adjacency matrix induced by DAG by formulating the generalized linear SEM as optimization problem in the framework of VAE.**

**Meeting of 28/05/2020**

Paper: The geography of COVID-19 spread in Italy and implications for the relaxation of confinement measures

Presenter: Martin Cepeda

Participants: all TAUers were there!

**Main idea of the article: **Using transport-based contact matrices between Italian regions, the authors were able to effectively model bed occupancy, disease spreading and possible post lock-down scenarios for all 107 Italian provinces.

Slides

**Meeting of 08/05/2020**

Paper1: Geometry-Aware Gradient Algorithms for Neural Architecture Search

Paper2: Meta-Dataset : a dataset of datasets for learning to learn from few examples

Presenter: Romain Egelé, Adrian El Baz

Participants: all TAUers were there!

**Main idea of the article 1:**

**The article focused on gradient-based neural architecture search with weight sharing. It is building a formal framework to understand the optimization of these methods. It is bringing an exponentiated gradient to accelerate the convergence to a stationary point. It is bringing a KL-divergence regularization to increase the sparsity of mixture-weights. The authors are also introducing a random search with weight sharing as a new baseline for neural architecture search. Finally the author benchmark existing methods on CIFAR-10 and tested an HPO algorithm ASHA on the NAS problem.**

**Main idea of the article 2: #TODO (@Adrian)**

Slides:

paper1: slides

paper2 **#TODO (@Adrian): please add link here**

**Meeting of 19/03/2020**

Paper: Evolving Neural Networks through Augmenting Topologies

Presenter: Saumya Jetley

Participants: all TAU was there!

**Main idea of the article: #TODO (@Saumya)**

Slides: **#TODO (@Saumya)**

**Meeting of 3/10/2019**

Paper: Each one's 3 favorite papers in ECML PKDD 201

Presenter: Guillaume Doquet, Michèle Sebag, Pierre Wolinski, Victor Berger, Zhengying Liu

Participants: all TAU was there!

**Main idea of the article: **

Slides: **#TODO**

**Meeting of 11/07/2019**

Paper: Liu, Hanxiao, Karen Simonyan, and Yiming Yang. "Darts: Differentiable architecture search." ICLR 2019

Presenter: Zhen

Participants: Zhengying, Zhen

**Main idea of the article: **Relaxation on the architecture used, like doing lasso on the architecture

Slides: **#TODO: @Zhen**

**Meeting of 20/06/2019**

Paper: Adam Gaier and David Ha, "Weight Agnostic Neural Networks". arxiv

Presenter: Zhengying

Participants: Lisheng, Zhen, Nathan

**Main idea of the article: **Learning / training ONLY the architecture of neural networks using neuroevolution (NEAT), with UNIFORM (or random) weight for ALL neurons.

**Meeting of 23/05/2019**

Paper: Chelsea Finn et al. "Online Meta-Learning". ICML 2019

Presenter: Lisheng

Participants: Zhengying, Heri, Zhen, Nathan

**Main idea of the article: #TODO: @Lisheng**

Slides **#TODO: @Lisheng**

**Meeting of 16/05/2019**

Paper: Lukas Hahn, et al. “Fast and Reliable Architecture Selection for Convolution Neural Networks”

Presenter: Adrien

Participants: Zhen, Nathan, Zhengying, Heri, Isabelle

**Main idea of the article: #TODO: @Adrien**

Slides **#TODO: @Adrien**

**Meeting of 02/05/2019**

Paper: Marcin Andrychowicz, et al. “Learning to learn by gradient descent by gradient descent”

Presenter: Nathan

Participants: Zhen, Loris, Zhengying

**Main idea of the article: #TODO: @Nathan**

Slides **#TODO: @Nathan**

**Meeting of 25/04/2019**

Paper: Haifeng Jin, et al. “Auto-Keras: An Efficient Neural Architecture Search System.”

Presenter: Heri

Participants: Guillaume, Isabelle, Lisheng, Nathan, Zhen, Zhengying

**Main idea of the article: **This paper introduces new metric of distance between two neural architecture. This metric is composed of layer wise distance and connection wise distane

Slides

**Meeting of 11/04/2019**

**#TODO: @Zhen**

**Meeting of 28/03/2019**

Paper: Franceschi et al. "Bilevel Programming for Hyperparameter Optimization and Meta-Learning". ICML 2018

Presenter: Zhengying

Participants: Heri, Pierre

**Main idea of the article: **Use a bi-level optimization formulation to unify Hyperparameter Optimization and Meta-learning, then use a smooth procedure (a sequence of gradient descent) to approximate the dependence of the solution of the inner problem (as in bi-level formulation).

**Meeting of 21/03/2019**

No specific paper. Slides on Automated Deep Learning are presented.

Presenter: Zhengying

Participants: Zhen

**Main idea of the article: **Formulate AutoML as two layers of learning: supervised learning + reinforcement learning.

**Meeting of 07/03/2019**

Paper: de Laroussilhe, Quentin, et al. "Neural Architecture Search Over a Graph Search Space." arXiv preprint arXiv:1812.10666 (2018).

Presenter: Guillaume Doquet

Participants: Heri, Zhengying, Lisheng, Marc, Guillaume C, Adrien, Zhen

**Main idea of the article: #TODO: @Guillaume**

Slides **#TODO: @Guillaume**

**Meeting of 31/01/2019**

Paper: Finn, Chelsea, Pieter Abbeel, and Sergey Levine. "Model-agnostic meta-learning for fast adaptation of deep networks." ICML2017

Presenter: Pierre

Participants: Heri, Zhengying, Lisheng

**Main idea of the article: #TODO: @Pierre**

Slides **#TODO: @Pierre**

**Meeting of 24/01/2019**

Paper: Sun-Hosoya et al: ActivMetaL: Algorithm Recommendation with Active Meta Learning (Related papers: paper1, paper2)

Presenter: Lisheng

Participants: Lisheng, Zhengying, Loris

**Main idea of the article: **Bring AutoML problem to recommender system, treat unknown performance as missing values in the meta learning matrix and use matrix factorization (non-probabilistic vs. probabilistic) techniques to solve that matrix.

**Meeting of 15/11/2018**

Paper: Falkner et al: BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Presenter: Heri

Participants: Michèle, Pierre, Loris, Guillaume.

**Main idea of the article: **Improve Hyperband approach by using Baysian Optimization for sampling the set of hyperparameter (instead of uniform sampling for standard Hyperband)

Slides

**Meeting of 25/10/2018**

Paper: Madrid et al: Towards AutoML in the presence of Drift: ﬁrst results

Presenter: Zhengying

Participants: Loris, Heri, Marc, Michèle, Guillaume Charpiat

**Main idea of the article: **In a lifelong learning + concept drift + AutoML setting, the authors use auto-sklearn + a drift detector + several model adaptation methods (e.g. re-train completely the model, update weights, add model, etc) and have some basic results for the AutoML3 challenge

**Meeting of 11/10/2018**

Paper: Andrew Brock, Theodore Lim, J.M. Ritchie, Nick Weston "SMASH: One-Shot Model Architecture Search through HyperNetworks".

Presenter: Guillaume Doquet

Participants: Loris, Heri, Zhengying, Pierre, Guillaume Charpiat

**Main idea of the article:** **#TODO: Guillaume**

Slides (**#TODO: Guillaume**)

**Meeting of 20/09/2018**

Paper: Lorraine, Jonathan, and David Duvenaud. "Stochastic Hyperparameter Optimization through Hypernetworks." arXiv preprint arXiv:1802.09419 (2018).

Presenter: Pierre

Participants: Michèle, Marc, Guillaume Charpiat, Heri, Zhengying

**Main idea of the article:** **#TODO: Pierre**

Slides (**#TODO: Pierre**)

**Meeting of 14/06/2018**

Paper: Linnan, Wang; Yiyang, Zao; Yuu, Jinnai "AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search".

Presenter: Heri

Participants: Michèle, Marc, Zhengying, Pierre

**Main idea of the article:** Use Monte Carlo Tree Search method to find optimal architecture for neural network. The proposed approach exploits the "block" design introduced in NAS (Neural Architecture Search). Another composant to speed up the search is the meta-DNN model (for prediction of the performance of a given architecture).

Slides

**Meeting of 07/06/2018**

**PhD seminar**

Paper : Wolpert, David H., and William G. Macready. "No free lunch theorems for optimization." IEEE transactions on evolutionary computation 1.1 (1997): 67-82.

Paper: Wolpert, David H. "The lack of a priori distinctions between learning algorithms." Neural computation 8.7 (1996): 1341-1390.

Presenter: Zhengying

Participants: Guillaume C., Lisheng, Victor B., Olivier, Diviyan, Théophile, Victor E., Giancarlo

**Main idea of the article:** Any two (optimization) algorithms work equally well when their performance is averaged across all possible problems.

**Meeting of 17/05/2018**

Paper : Al-Shedivat, Maruan, et al. "Continuous adaptation via meta-learning in nonstationary and competitive environments.". ICLR 2018.

Presenter: Lisheng

Participants: Zhengying, Heri, Guillaume D.

**Main idea of the article:** Learn (via optimizing over-task loss) to continuously adapt policy to nonstationary (modeled as competitive agents) environments. Solve purely RL problems.

**Meeting of 03/05/2018**

Paper : Lisha, Li, et al. "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization". ICLR 2017.

**# TODO: Aris**

Presenter:

Participants:

**Main idea of the article:**

Slides

**Meeting of 12/04/2018**

**Evolving the Topology of Large Scale Deep Neural Networks **

Presenter: Guillaume DOQUET

Participants: Isabelle, Marc, Michèle, Laurent, Aris, Heri, Pierre, Guillaume, Zhengying

**Main idea of the article:** Use an evolutionary strategy to learn the structure of a deep neural network. The algorithm operates on 2 levels simultaneously : the macro scale (number and type of layers) and the layer scale

(parameters of that layer). Results are presented on CIFAR10, CIFAR100, MNIST, and Fashion-MNIST, beating or rivalizing state-of-the-art performance.

Slides

**Meeting of 05/04/2018**

**Large-Scale Evolution of Image Classifiers**

**Learning Transferable Architectures for scalable image recognition**

Presenter: Pierre Wolinski

Participants: Zhengying, Guillaume Charpiat, Heri, ?

**Main idea of the articles:**

1st article: basic evolutionary algorithm;

2nd article: train a RNN to build a convolutional neural network, where blocks of layers have the same structure (inception-like networks).

Slides : 1st article, 2nd article

**Meeting of 29/03/2018**

**Monte Carlo tree search for algorithm selection (MOSAIC)**

Presenter: Heri

Participants: Zhengying, Guillaume (Charpiat, Doquet), Pierre, Aris

**Main idea of the article:** Tackle hyperparameter optimization problem using monte carlo tree search: state (value of hyperparameters already choosed), action (value of the next hyperparameter to choose), reward (CV-score). The algorithm is composed of two parts: bandits part (designed for algorithm selection: random forest, svm, ...) and MCTS (for preprocessing and algorithm configuration). This new idea produces good (first) results but many improvements still to be done.

Slides

**Meeting of 22/03/2018**

**Synthesis of AutoML Reading Group (21/11/2017-22/03/2018) **(yes with color! and you can do that too!)

Presenter: Zhengying

Participants: Isabelle, les 2 Guillaumes, Heri, Pierre, Lisheng

**Main idea of the article:** Describe the AutoML problem in a comprehensive and intuitive optimization manner, formulate many existing AutoML approaches in a uniform way, attach each approach to one step in the classic machine learning pipeline and make some discussion on future research ideas.

**Meeting of 08/03/2018**

Paper : Swersky, Kevin, Jasper Snoek, and Ryan Prescott Adams. "Freeze-thaw Bayesian optimization." arXiv preprint arXiv:1406.3896 (2014).

Presenter: Lisheng

Participants: Isabelle, Marc, Guillaume Charpiat, Heri, Pierre, Zhengying

**Main idea of the article:** A strategy for efficiently choosing hyperparameters: pause the training of models that are not promising. Model training curves as samples of Gaussian process.

**Meeting of 01/03/2018**

Paper : M. Feurer, A. Klein, K. Eggensperger, J.T. Springenberg, M. Blum, F. Hutter "Efficient and Robust Automated Machine Learning" (NIPS 2015)

Presenter: Guillaume Doquet

Participants: Pierre, Zhengying, Isabelle, Guillaume Charpiat

**Main idea of the article:** Combine meta-learning, tree-based Bayesian Optimization (SMAC) and ensemble method (ensemble selection) to tackle AutoML problems.

**Meeting of 22/02/2018**

Paper: Frank Hutter, Holger H. Hoos, Kevin Leyton-Brown (*International Conference on Learning and Intelligent Optimization* 2011)

Presenter: Pierre

Participants: Zhengying, Guillaume D, Guillaume C, Lisheng, Isabelle

**Main idea of the article**: Presentation of the hyperparameter search algorithm SMAC. SMAC is a SMBO-based algorithm using random forests to model hyperparameters. Moreover, it implements the case where one tunes the hyperparameters for multiple *instance sets*. (Slides)

Questions & Remarks: *instance set* possibly refers to a split of a data set between a train set and a validation set

**Meeting of 08/02/2018**

Paper: R. Bardenet, M. Brendel, B. Kégl, M. Sebag "Collaborative hyperparameter tuning" ICML (2013).

Presenter: Heri

Participants: Zhengying, Isabelle, Aris, Pierre, Heri

**Main idea of the article:** By collaborative tuning of hyperparameter on multiple datasets, one can incorporate (expert) knowledge from similar tasks to improve Bayesian hyperparameter search. Hyper-parameter ranking is used (instead of validation score) to assess the quality of one hyperparameter. (slides)

**Meeting of 01/02/2018**

Paper: Liu, C., Zoph, B., Shlens, J., Hua, W., Li, L. J., Fei-Fei, L., & Murphy, K. "Progressive neural architecture search." arXiv preprint arXiv:1712.00559 (2017).

Presenter: Aris

Participants: Guillaume Charpiat, Guillaume Doquet, Heri, Lisheng, Zhengying, Aris

**Main idea of the article:** Learn an RNN that estimates the quality of a CNN sub-module ("cell") generated using multiple blocks (each chosen from some fixed options of convolutions and pooling operators).

When expanding the cell structure, the RNN is used to prune the search space. (Slides)

**Meeting of 25/01/2018**

Paper : Max Jaderberg, Karen Simonyan, Andrew Zisserman and Koray Kavukcuoglu. "Spatial transformer networks." arXiv preprint arXiv:1506.02025v3 (2016).

Presenter : Lisheng

Participants : Michèle, Guillaume Charpiat, Aris, Heri, Guillaume Doquet

**Main idea of the article : ** A Spatial transformation network which learns an appropriate transformation of input feature map, is proposed to be inserted to existing architecture to make the task (e.g. classification) in later layers easier, this is possible mainly because the STN is differentiable. (Slides)

To-do:

Idea for potential use: A spatial transformer network can be viewed as a data generator constrained by final performance of the entire network.

**Meeting of 18/01/2018**

Paper : Koutník, Jan, Juergen Schmidhuber, and Faustino Gomez. "A frequency-domain encoding for neuroevolution." arXiv preprint arXiv:1212.6521 (2012).

Presenter : Zhengying

Participants : Michèle, Isabelle, Guillaume Charpiat, Lisheng, Aris, Heri, Guillaume Doquet

**Main idea of the article : ** Solve Octopus Arm Problem by using a few Fourier coefficients (chromosome) to compactly represent recurrent neural networks and using Natural Evolution Strategy to select promising prior distribution of neural networks. (Slides)

To-do:

Heri and Zhengying will run experiments following Michèle's idea

**Meeting of 11/01/2018**

Paper : Ravid Shwartz-Ziv and Naftali Tishby, "Opening the Black Box of Deep Neural Networks via Information" (Arxiv, March 2017)

Presenter : Guillaume Doquet

Participants : Guillaume Charpiat, Cyril, Zhengying, Heri, Guillaume Doquet

**Main idea of the article : **Deep neural networks go through 2 distinct phases during training. In the first phase, the mutual information between each hidden layer and the labels increases. In the second phase,

the mutual information between the layers and the data decreases. In other words, a compressed latent representation of the data is found. This is a byproduct of the stochastic nature of the gradient descent. (Slides)

**Meeting of 04/01/2018**

Paper: Saxe, Andrew M., et al. "On Random Weights and Unsupervised Feature Learning" (ICML 2011).

Presenter: Pierre Wolinski

Participants: Michèle, Guillaume Doquet, Guillaume Charpiat, Zhengying, Heri, Aris

**Main idea of the article**: In some cases, untrained neural networks are almost as accurate as trained neural networks. By studying the Fourier transform of the convolution, we are able to explain these results. Moreover, the article gives a heuristic for architecture selection. (Slides)

To-do:

Compare the eigenvalue distribution of a random filter and a trained filter

Replace the random layer applied over an image with the Fourier transform of the image => better ? worse ?

**Meeting of 21/12/2017**

Paper: Domhan et at. "Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks by Extrapolation of Learning Curves" (IJCAI 2015).

Presenter: Heri

Participants: Michèle, Guillaume (Doquet, Charpiat), Isabelle, Pierre, Zhengying

**Main idea of the article**: Speeding up the search of hyperparameter by predicting the performance of the model. Use a set of parametric function to extrapolate the learning curve. Stop runs that are unlikely to perform the best observed so far. (Slides)

To-do:

Run the code on new dataset

**Meeting of 30/11/2017**

Paper: Klein, Aaron, et al. "Fast bayesian optimization of machine learning hyperparameters on large datasets." *arXiv preprint arXiv:1605.07079* (2016).

Presenter: Zhengying

Participants: Isabelle, Michèle, Marc, Lisheng, Guillaume (Doquet), Heri

**Main idea of the article**: Use Bayesian Optimization to do hyperparameter selection, with faster training (thus faster loss evaluation) using sampled sub-dataset, following a strategy that chooses next point to evaluate by maximizing information gain per computational cost on the distribution of the global minimum of the goal function (e.g. validation error w.r.t hyperparameter)

Slides: 10 pages, contains also a very brief introduction to Bayesian Optimization and Gaussian Process, with a small exercise ;)

Remarks & questions:

In Bayesian Optimization, the Bayesian philosophy is applied, but Bayes theorem is not used

Isabelle and Lisheng are using an approach with similar idea (maximize knowledge gain per computational cost) and which is even more general

To-do:

Zhengying will run the code of the authors to test the performance of the algorithm

**Meeting of 21/11/2017**

Paper: Munoz, Mario A., et al. "Instance Spaces for Machine Learning Classification." *Mach. Learn* (2017).

Presenter: Guillaume Doquet

Participants: Michèle, Lisheng, Heri, Zhengying

**Main idea of the article**: Extend the Algorithm Selection Problem framework suggested by Rice to gain knowledge on how well the combination of different algorithms and datasets can be or to objectively measure the performance of an algorithm, using 2-d visualisation in a so-called instance space. For each instance (a dataset and a classification problem), a lot of features are computed and then selected. A performance is adopted then SVM is used for fitting.