Embedded Files
Hi! I'm Alex Levine, a Postdoctoral Fellow working with Amy Zhang (ECE) and Peter Stone (CS) at The University of Texas at Austin. I completed my Ph.D. in August 2023 in Soheil Feizi's group at the University of Maryland Department of Computer Science.
My current research focus is in reinforcement learning: my interests include interpretable deep representation learning for control tasks; robust reinforcement learning; and goal-conditioned reinforcement learning. During my Ph.D., I worked on adversarial robustness in supervised deep learning, and proposed several novel techniques for classification with deterministic robustness guarantees at ImageNet scale.
Feel free to reach out to me via email at alevine0 _at_ cs _dot_ utexas _dot_ edu, or via LinkedIn.
Read more about my work below:
Reinforcement Learning
Reinforcement Learning
Representation Learning for Reinforcement Learning
Representation Learning for Reinforcement Learning
In order to train agents that can quickly adapt to new objectives or reward functions, efficient unsupervised (i.e., reward-free) learning can be important in sequential decision-making tasks. In particular, in real-world control settings, the observation space is often noisy and unnecessarily high-dimensional. A more compact state representation can then be learned, which may then be used to solve many downstream tasks more efficiently than would be possible using "raw" observations directly. One formalization of such settings is the Exogenous Block MDP (Ex-BMDP) model (Efroni et al. 2022). In the Ex-BMDP formalization, observations can be factorized into an action-dependent latent state which evolves deterministically, and action-independent time-correlated noise. The objective of the representation learning algorithm is to learn an encoder that maps from observations to the controllable latent state representations, while removing the exogenous noise. In the following works, I developed algorithms for correct and efficient representation learning in the infinite horizon, no-reset variant of Ex-BMDP model:
Efroni et al. (2022) introduces a provably sample-efficient algorithm, PPE, for learning a controllable state encoder in the Ex-BMDP setting. However, PPE is designed only for the finite-horizon, episodic version of this setting, where the controllable latent state resets to a single specific start state after each episode. In this work, we propose STEEL, the first provably sample-efficient algorithm for learning the controllable representation of an Ex-BMDP from a single trajectory, with no ability to reset the latent state. STEEL has a sample complexity that depends only on the sizes of the controllable latent space and the encoder function class, and (at worst linearly) on the mixing time of the exogenous noise factor: its sample complexity does not explicitly depend on the size of the exogenous noise factor.
Lamb et al. (2022) proposes a practical and straightforward-to-implement (although not provably sample-efficient) algorithm, AC-State, for learning a controllable latent representation of an Ex-BMDP in the infinite-horizon setting. AC-State works by learning to predict the first action along a path, given the encodings of the initial and final observations in the path. However, in this work, we show that AC-State will fail to learn a correct latent representation under certain types of dynamics, even in the limit of infinite environment steps. We then propose ACDF, a corrected variation of AC-State which is guaranteed to (eventually) learn a correct encoder in these settings.
Goal-Conditioned Reinforcement Learning
Goal-Conditioned Reinforcement Learning
Many applications of reinforcement learning can be formalized as goal-conditioned environments, where, in each episode, there is a "goal" that affects the rewards obtained during that episode but does not affect the dynamics. In this work we propose a new technique for reinforcement learning with high-dimensional goals. Specifically, we explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. We use this connection to apply Gradient-Based Attention Transfer (Zagoruyko and Komodakis 2017), a knowledge distillation technique, to the Q-function update. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional.
Certifiable Robustness
Certifiable Robustness
Standard deep-learning classifiers are known to be highly sensitive to small changes in their inputs, including adversarially-constructed distortions which may be imperceptible to human inspection. A certifiably robust classifier is a special type of classifier that aims to solve this problem, by providing sample-wise guarantees on its output sensitivity. While a standard classifier's output (given input x) can be interpreted as simply meaning "Sample x belongs to class c", a certifiably robust classifier produces an additional output, the certified radius r, which indicates "Sample x belongs to class c, AND no change to x of magnitude less than r will cause this classifier to classify x differently." In the papers below, I proposed various novel techniques for certifiably robust classification, with improved scalability and determinism:
We proposed a novel algorithm for certifiably robust classification, with the certified radius measured in the L1 norm. Previous works on certified robustness against Lp-norm distortions either (a) required Monte-Carlo sampling, and therefore produce only probabilistic, rather than exact, certificates, or (b) failed to scale to large-scale tasks (i.e. ImageNet classification). Our algorithm has neither limitation, providing deterministic certification at ImageNet scale.
We extended the work discussed above to certify robustness against distortions in an Lp radius, for any p<1. This provides the first technique to our knowledge for certifying robustness under Lp metrics for p ∈ (0,1); provides deterministic certificates; and scales to ImageNet. We also (in the appendix) provide a deterministic variant of a technique we proposed previously for L0 certification, with similar performance.
We provided a (deterministic, large-scale) certifiably robust image classifier with robustness guarantees against patch adversarial attacks: The output of the classification will not change under an arbitrary distortion to any square patch of the image smaller than a bounded size.
Most certifiably robust classifiers provide guarantees for consistent output under distortion of the input sample x. In this work, we instead developed methods for certifiably robust classification which guarantee consistent output under distortions to the training data, i.e., (i) a bounded number of training sample label changes and (ii) a bounded number of training sample insertions/deletions. In the former case, we significantly outperformed prior methods.
We introduced a novel algorithm for certifiably robust classification under sparse (L0) distortions: in the context of image classification, the size of the distortion is measured in terms of the number of pixels which have been modified. Our method significantly improved over the prior state of the art for L0 certifiably robust classification.
Most work on robustness certification focuses on robustness to Lp-bounded distortions; however, these may not be the most meaningful metrics for measuring changes in inputs in all domains. In this work, we developed a certifiably robust image classifier that is robust under distortions in the Wasserstein metric (or "earth-mover distance"), where local shifts in pixel intensity between nearby pixels are considered smaller distortions than longer-range shifts.
Page updated
Report abuse