[ paper ] 

• Uses model-generated rationales to discover the skills tested per test sample

• Grouping instances by skill reveals stark differences in performance of leading models from OpenAI, Google, and Anthropic

[ paper ]

• Introduces ArtSavant, a practical tool inspired directly by legal literature, to help artist's argue cases of copyright infringement

• Features a novel interpretable classification method based on matching of tag signatures

• Formulates 'style copying' from the lens of classification over image sets

[ paper ]

• Identifies CLIP text encoder as a key bottleneck in compositional generation, using optimization and attention contribution approaches

• Proposes a simple project approach to improve compositional generation via linear projection of CLIP embeddings

[ paper ]

• Essentially a VLM cookbook; accessible read on current state of VLMs and open problems

[ paper; updated version and code soon ]

• We leverage WorldBank data to see how well LLMs can answer the same question about different countries

• We observe pervasive performance disparities across regions and income groups for 20 cutting edge LLMs

• I got to write this paper with my mom!! ;)

[ paper; updated version and code soon ]

• We uncover a fairness issue with typical Zero-shot classification

• We devise a novel classification paradigm that explicitly accounts for diverse subpopulations within classes via (i) LLM-inferred attributes and (ii) a non-linear consolidation scheme

• Our method features transparency for free (at least as performant as strong baselines), while improving fairness (reduced disparities)

[ paper ]

• Tag-based method to generate faithful descriptions of failure modes

• Features an efficient search to obtain minimal tag combinations that result in greatest loss of accuracy

[ paper ]

• Scalable method for sorting one's data by spuriosity, revealing examples of relevant spurious cues and natural counterfactuals within one's dataset

• Bias seems to have more to do with data than training. By sorting our data, we can get much more out of it, like easy bias measurement + mitigation.

[ paper ] [ Lens website -- I highly recommend visiting! ]

• Automatic Concept Extraction is dictionary learning! And, according to an extensive suite of complementary metrics, NMF is most often best

• Check out Lens 🔎 for visualizations of concepts extract from ImageNet

[ paper ] 

• Rethinking model training workflow: can we debug models via perceptual similarity search on large pools of unlabeled data?

[ paper ] [ video presentation ] [ code ] 

• Efficient method to extend VLM capabilities to vision-only models; only requires training one linear layer with a small amount of unlabeled images)

• Many interpretability benefits: concept bottleneck models w/o concept supervision, diagnosing distribution shifts, improved image retrieval

[ paper ]

• We observe that a linear layer can map diverse vision representation spaces, and then use CLIP's text encoder to get concept activation vectors for arbitrary vision models without needing any exemplar data 

[ paper ] [ dataset webpage ] 

• How should models treat objects that often appear i. small, ii. uncentered, iii. and alongside spurious cues that are much easier to recognize? Perhaps single class-labels are insufficient... 

• We offer segmentations, image rankings, and analysis for 15 classes for which models heavily rely on spurious cues

[ paper ] 

• Key finding: adversarial training can increase reliance on spurious features, which reduces distributional robustness.

• Theoretical analysis on effect of adv. training norm + scale of spurious feature,  along with extensive empirical evidence over 5 diverse benchmarks.

[ paper ] [ dataset webpage ] 

• Salient Imagenet-1M dataset: millions of segmentation masks for core and spurious features, 5000 core and spurious features discovered

• Novel training paradigm  and analysis of 42 diverse pretrained models

[ paper ] [ dataset webpage ] 

• RIVAL10 dataset: segmentations for object and 18 visual attributes

• Quantitative noise and saliency based analyses

• Selected for Oral Presentation (top 4.2%)

[ paper ] 

• Clean and Adversarial distributions nearly linearly separable in SimCLR space (77% accuracy for linear classifier trained on only two samples)

• Extensible to evasion, unseen, poisoning, and adaptive attacks

[ paper ]

• Self-Attention Cycle GAN approach to Genre transfer with interpretability analysis and human-evaluation based metric development