This plot illustrates the prior we use in Condensa when searching for the global sparsity value, By default we setup a simple 1D Gaussian process. In this plot above, you will notice 3 past observations. The solid black line is the GP surrogate mean prediction of the objective function given the data, and the shaded area shows the mean plus and mean minus the variance. The superimposed Gaussians corresponds to the GP mean and standard deviation of the prediction at the 3 previously sampled points.

Here there are 2 plots, the plot on the left illustrates Condensa sparsity profile for ResNet 110 (CIFAR-10) Filter Pruning and the Goal for Bayesian optimization is search for a global sparsity value that satisfies a user provided accuracy constraint, for example line 17 in the listing above (- 2.00%).

The plot on the right, illustrates how we modeled our prior belief of this expensive black-box function (L-C , the Green Curve) to be a Gaussian Process, with kernel functions that have default values, and can be easily modified by the Condensa user, depending on the dataset-network under compression.

Gaussian Processes (GP) are a generic supervised learning method designed to solve regression and probabilistic classification problems.

The advantages of Gaussian processes are:

• The prediction interpolates the observations for regular kernels.
• The prediction is probabilistic (Gaussian) so that we can compute empirical confidence intervals and decide based on those if one should refit the prediction in some region of interest.
• Versatile: different kernels can be specified. Common kernels are provided, in Condensa but it is also possible to specify custom kernels.

The choice of co variance functions for the Gaussian Process is crucial, as it determines the smoothness properties of samples drawn from it.

• The squared exponential kernel is naive, in that divergences of all features of sparsity affect the co-variance equally.
• In Condensa, we use the Mat´ern [Mat´ern, 1960, Stein, 1999] kernel by default, which incorporates a smoothness parameter, ς to permit greater flexibility in modeling functions.
• The Mat´ern kernel reduces to the squared exponential kernel as ς tends it infinity and when ς = 0.5, it reduces to the unsquared exponential kernel.
• While the squared exponential and Mat´ern are the most common kernels for GPs. "Appropriate covariance functions can also be used in Condensa to extend the model in other interesting ways"

• One intuitive strategy is to maximize the probability of improving over the best current value (incumbent).
• This function is also sometimes called MPI, for “maximum probability of improvement” or “the P-algorithm”,since the utility is the probability of improvement.
• A somewhat more satisfying alternative acquisition function would be one that takes into account not only the probability of improvement, but the magnitude of the improvement a point can potentially yield. In particular, one could minimize the expected deviation from the true maximum, when choosing a new trial point.

• A more recent development is the idea of exploiting lower confidence bounds (upper, when considering maximization) to construct acquisition functions that minimize regret over the course of their optimization.
• Like other parameterized acquisition models we have seen, the parameter κ is could be decided by user.
• PI, EI and GP-UCB give rise to distinct sampling behaviour over time, as we illustrate in the figure below.
• With several different parameterized acquisition functions in the literature, it is often unclear which one to use. For the problem of Automated DNN model compression which is the goal of Condensa, we propose ILS-UCB, that is described next.

• In addition to unconstrained optimization, to enable CONDENSA to achieve constraint satisfaction we build on top of level-set black-box optimization (Bogunovic et al., 2016; Garg et al., 2016; Zanette et al., 2018).
• We leverage a Gaussian Process Adaptive Sampling criterion called Implicit Level Set Upper Confidence Bound (ILS-UCB) (Garg et al., 2016), that prioritizes sampling near a level set of the estimate.
• This model prioritizes searching the expected L-C curve intersection with user accuracy constraints, conditional on estimated uncertainty, and does not seek to precisely learn the shape of the entire L-C curve.

• Intuitively, by reducing the estimation space to specifically localize the sparsity that meets user accuracy constraints, we can "Reduce the total number of measurements"
• Consequently the "time required to achieve an optimal value for the sparsity".
• Hence, rather than prioritizing both high variance and high mean like UCB, ILS-UCB prioritizes sampling in areas near a level set of the mean represented by the Gaussian Process Implicit Surface.
• We propose to minimize the implicit potential defined by µ(x) − L, and where the confidence interval is large:

• On three real-world image classification and language modeling tasks, CONDENSA achieves memory footprint reductions of up to 65× and runtime throughput improvements of up to 2.22× using at most 10 samples per search.
• With the initial framework in place, we envision a number of directions to expand on CONDENSA’s capability. For example, we plan to "augment automatic sparsity ratio inference with support for additional compression hyperparameters such as block sizes in block-sparsification" (Gray et al., 2017), and data types for quantization.
• Our long-term goal is "a framework that makes model compression easier, more flexible, and accessible to a wide range of users".