Concerning the slide with the smooth activation functions providing better security: how reproducible is this on other data-sets?

The dataset in the slide 50 is CIFAR-10. This work https://arxiv.org/pdf/2006.14536.pdf showed an improvement for the ImageNet dataset trained on the ResNet-50 architecture (Fig. 3).

When you talk about the trade-off between labelled and unlabeled data in adversarial training, and find the best ratio to be 3:7, does this hold across data-sets?

The ratio of labeled-to-unlabeled data per batch should be tuned per dataset and model. As a rule of thumb, we usually have more unlabeled data and should give proportionally more weight to this data.

Can you provide more details on 3D printing for adversarial attacks. You mentionned the EOT technique but it seems unclear how an attack on an image is used to modify the 3D model to be printed.

By introducing EOT, a general-purpose algorithm for creating robust adversarial examples, and by modeling 3D rendering and printing as a transformation within the framework of EOT, the authors succeeded in fabricating three-dimensional adversarial objects. The key point is to differentitate through the rendering process. Please, check Section 2.2 in http://proceedings.mlr.press/v80/athalye18b/athalye18b.pdf

(from [GA21]'The closer to boundary, the larger weight' ). As you mentioned some methods for adversarial defense use importance sampling on adversarial data. Are you aware of any study that measures the impact of these methods on the uncertainty of the model on rare cases ?

I am not aware of any study that measures impact of the importance sampling on the uncertainty of the model on rare cases. It sounds like a good idea that is worth trying.

(from [GQ20]). Given those two claims: "larger models are more robust" but "robust generalization [and larger models]requires more data", what is your feeling towards the best practice data scientists should adopt for safety critical ML systems ?

I would recommend: (1) training models with high-capacity from scratch, (2) curating the data before training, (3) using the state of the art defense techniques.

(from "Channel-wise activation magnitude"). The correlation of the magnitude of some channels and adversarial attacks have been observed for PGD attacks. Has it been confirmed for other attacks as well ? Could it be used as an efficient safeguard to detect adversarial attacks ?

In this work [IA21], the robustness (accuracy on adversarial examples) is evaluated under attacks: FGSM [GS15], PGD-20 [MM18], and CW1 [https://arxiv.org/abs/1608.04644] optimized by PGD (Section 5.1 in [IA21]). In my opinion, the method is yet to be tested in terms of robust defense. However, it gives us an insight into internal mechanisms in neural networks that are activated during the attacks.

Is the metric of robustness always performed with PGD ? Is there any work using formal deterministic methods or the CLEVER score to measure the robustness to adversarial attacks ? It seems to me that the robustness score should not be biased by the choice of heuristics used to created adversarial attacks.

No, PGD is a non-adaptive white-box attack and is not strong enough to assess robustness. The adaptive white-box attacks in a way described in https://arxiv.org/pdf/2002.08347.pdf are the state-of-the-art methods.

(from [YG19] high accuracy on high frequency) On what data (low/high freqsuency) the model has been trained on ?

The results in my talk were presented for the ImageNet dataset. The authors of the paper also used CIFAR-10 dataset as well as datasets with common corruptions, e.g. ImageNet-C

https://arxiv.org/abs/1807.01697 (images with fog, snow, etc.).

Regarding the open questions at the end of the talk: many of the listed open questions are problems that people worked on for the last 3-5 years (e.g., why adversarial examples exist). In your talk it sounded a bit like these problems are still as "open" as before. From reviewing all this literature, in your opinion, what are the interesting/promising directions to continue working on to addresse these problems? I would be particularly interested in the problem of why adversarial examples exist and, in terms of robustness, whether adversarial training based defenses will solve robustness or whether we need to rethink our strategy.

There are open problems and progress has been made, however, we do not have fully satisfying answers as yet. The community actually lacks consensus on this point: Szegedy et al. [BS14] suggest that neural networks have blind spots, Goodfellow et al. [GS15] posit that linearity is the main culprit and quantized inputs can break up linearity, and Xu et al. [https://arxiv.org/abs/1704.01155] claims that quantization makes the adversarial search space smaller. Personally, I like the visualization presented by Roth et al. [RK19], it intuitively shows that adversarial examples are caused by small measure regions of adversarial class “jutting” into a correct decision region. We can imagine that such a situation might occur in a high dimensional space. This model is commonly accepted because the inputs remain correctly classified when perturbed with small random noise (e.g., Uniform or Gaussian). However, we can also easily find the attack vector that moves the inputs to decision regions where they become misclassified. Up to this point, we have focused on models being the main culprit of why the adversarial examples exist. Mądry et al. shifted the gears and analyzed the adversarial examples from the dataset perspective, they defined the non-robust features that neural networks exploit to achieve a high accuracy classification. In my opinion, currently, adversarial training is the best empirical defense but the robustness that it provides is far from the desired one (for instance, up to 60% while the accuracy on clean data is already about 99%). I suspect that the promising directions are to design new model architectures and control the features that are utilized by neural networks during training.

You mentioned that the ideal case of fixing a non-robust model is to add some kind of patches to it. Could you elaborate in more detail what patches do you mean here? Would techniques like adversarial training qualify as patches? I'm asking because for me it seems that by requiring adversarial robustness (say, with respect to small Lp-bounded perturbations), we change the classifier quite dramatically, in particular its standard accuracy, privacy properties, local smoothness, etc. In other words, by changing the objective from a standard loss minimization to robust loss minimization, we are essentially looking for a completely different classifier which wouldn't qualify as a patch from this point of view.

By patching I mean the principle of strengthening a trained classifier for improved adversarial robustness, instead of training the model again from scratch. You are right that if one uses techniques like adversarial training on a standard trained model, their loss landscapes are too different to be meaningful. However, as we discovered in our recent AAAI paper <https://arxiv.org/abs/2012.11769>, we developed efficient and attack-agnostic training method to achieve this goal and make patching more practical. Similar, for mitigating backdoor effects, we also used model-connectivity based principle to "sanitize" the model with few clean data. See our ICLR'20 paper <https://openreview.net/forum?id=SJgwzCEKwH>.

One of the points of your roadmap is certification. What would be your opinion on what are the right perturbation sets to certify in the first place, and moreover how to determine their magnitude which is of practical interest? E.g. even if we are interested in being robust to small Linfinity perturbations, it's usually unclear what is the right Linfinity radius that we really want to be robust in. Of course, this aspect is very application-dependent, but I'm curious to hear your thoughts on this, maybe based on your experience with practical deployment of robust models.

This is a great question. I'd use the analogy from collision test for car models to motivate the answer. In car model development, we care a lot about safety, and there have been many safety reports, collision tests, and "guarantees" being reported. If we look closely, those numbers come with pre-assumptions (car driving at a certain speed; environment condition, etc). So I'd say instead of setting one robustness threshold and putting endless debate on this value, we should do more education on AI model developers and users to prepare them with the mindsets of how to use the model in a robust manner, just like the driver would not expect any collision guarantee at the spped of 300 km/hr, but the driver would expect the car to be "safe" at the speed of 60 km/hr. Similarly, AI model is not possible to be robust to unbounded perturbation. We have to standardize the notion of "normal mode" for model use.

How do you compute your CLEVER score? Is it based on a sampling around the data point or is it computed formally? Also. is this score computed for each. data. sample or are you able to derive a global score for the model?

Similar to randomized smoothing, we compute the gradient norms of data samples nearby a given data input, and use those sampled values to evaluate the local cross Lipschitz constant associated with our derived lower bound on minimal perturbation. So the CLEVER score is a local robustness metric estimating the distance to the closest decision boundary. Extending to global robustness, one can take data samples from different classes, compute their CLEVER scores, and use the statistics to indicate global robustness. A generic CLEVER score computation wrapper is provided by ART <https://adversarial-robustness-toolbox.readthedocs.io/en/stable/modules/metrics.html#clever>

AI is worth using if it performs better than other alternatives. Do you think that reducing accuracy to increase robustness might make the AI not worth using anymore?

From the standpoint of practical deployment, if one cannot guarantee robust generalization from in-house model development and testing to real-life problem solving in the wild, I am concerned that the continuous improvement in accuracy on a pre-defined test set may give us a false sense of technological breakthrough, especially when one needs to trade robustness for improved accuracy. That being said, there are low-risk and high error-tolerance uses cases where AI models without rigorous guarantees and validations are still valuable - such as automated question-answering, recommendation system, speech processing and so on. I'd say for high-stakes decision making tasks we need to be more cautious about the AI model. But there are certainly many other scenarios where introducing AI models are certainly beneficial.

Can you tell us more on the AI incident database and its future goals and derived applications ?

There is a very nice article on it: https://bdtechtalks.com/2021/01/14/ai-incident-database/

You talked about robustness in the context of larger systems and also mentioned model patching on the slides. I am wondering how important the ability to patch an existing model is in practice/in industry and whether we as a research community are somehow missing this application by mostly providing adversarial training based defenses (which usually need to be trained from scratch)?

In some cases clients want the service provider to find and train the best model on their behalf (e.g. AutoAI), then it's reasonable to invoke adversarial training and train from scratch. However, we also encounter scenarios that clients already have a preferred and trained model but want to understand better about the model properties, such as fairness, explainability, robustness, and so on. While inspecting and strengthening their model, the client may not want to drastically change the model significantly so as to remain the model utility (but granted that training from scratch with a different approach could be a solution if the utility wouldn't change much). I do feel this "on-demand" model hardening is inspired by practical needs and has not received much attention in the research community. In our recent AAAI'21 paper <https://arxiv.org/abs/2012.11769>, we also proposed a new on-demand training method that can bridge this gap while standard adversarial training methods fail to do so.

Can you provide more details on your 2020 IJCAI paper ? In particular, why using FSC to explain RNNs ? Could you have used other explanation heuristics ? How do you force the model to have understandable explanations ?

The idea was that FSCs provide a good formalism to 'trade off' performance of the policy (originally given by an RNN) and the complexity of the controller. By the integrated model checking, we have the means to assess the performance of the extracted controller, and can likewise control the number of states in the FSC.

The examples shown in this talk is a simple grid environment. However, in practice the environment model can be much more complicated, e.g., with an infinite number (continuous) of states or actions. Is it possible to handle these cases?

I used these examples to explain the concepts, in our numerical evaluation, we have a large range of practical examples, for instance within aircraft collision avoidance and spacecraft motion planning. Moreover, there are multiple industrial partners we currently work with, for instance in the area of predictive maintenance.

What kind of input perturbation are you using ? Is the set of sensors represented by the outputn probablities of a single neural network or by an ensemble of neural networks ? How did you design the reasoning model (based on ontology ?)

The perturbation can be general, in terms of L_p bounded or unrestricted for, say, image as an example. Basically the perturbations are vectors drawn from the same domain from the true data trying to mislead the prediction of machine learning models given the perturbed data.

It is a set of models (neural networks) for different tasks. For instance, taking the road sign recognition as an example, some sensors in charge of identifying the shapes, some for the contents etc. The design of the reasoning model is based on commonsense knowledge represented as first order logic rules, and using graphical models such as markov logic networks or Bayesian networks to embed them.

It seems that the more structure you put in your knowledge (hierarchies, relationships, etc.) the more useful it will be to eliminate adversarial examples (that will not match the structure). Is it correct?

Yes. If the structure of the knowledge is clear, the first order logic rules will be clear and therefore the graphical model can be constructed and we can make inference based on it.

You mentioned unrobust assignments and the need for more robust knowledge. How do you discover consistent (e.g. truck and not-animal) but incorrect predictions? Are you using some regularization encouraging diversity between the different underlying networks (e.g. similar to DVERGE https://arxiv.org/pdf/2009.14720.pdf)?

The sensing-reasoning is different with ensemble models such as DVERGE did which leverages statistical features to construct ensemble models. The knowledge integrated ML pipeline aims to use commonsense knowledge to help identify conflicts and improve the robustness of ML predictions. Indeed, if there is an attack that could attack every sensors such that all the rules are satisfied, the pipeline would be attacked. However we believe that's very hard, and if that can be done, maybe that instance is already a "true" sample instead of an adversarial instance.

Is it possible to run adaptive attacks on your robust models?

Adaptive attacks are mostly used for adversarial defenses based on certain heuristics. Our approach is based on a maximin robust regularizer, which works in a manner similar to adversarial training. Thus there is no obvious adaptive attack in this setting. Additionally, in our recent paper (https://arxiv.org/pdf/2101.08452.pdf) we propose the optimal adversarial attack, which is significantly stronger than other attacks and can theoretically find the worst case adversary. In that setting, our robustly trained agents still remain robust under this attack, showing that they are truly robust.

Could you give some details on the training time of your models? What is the price to pay to learn your most complex policy compared to other methods you compare with?

The training time for state-adversarial regularized agents (SA-PPO, SA-DDPG and SA-DQN) are 3x to 5x slower than training a vanilla agents. When alternating training with a learned adversary (ATLA) is used, it requires more iterations to converge due to the existence of an adversary, and the training time is 5x to 20x slower than training a vanilla agent. However, even for the classification setting, PGD based adversarial training also needs an order of magnitude more time to train (PGD steps are slow, and more epochs are needed). There is always a cost to pay to make a ML system robust, and our robust RL training procedure has a cost similar to those classification settings. We believe there is still a large room to improve the training efficiency for robust reinforcement learning.