Video-to-video synthesis models struggle with consistent character generation, smooth temporal transitions, and quality during fast motion. Joint fully cross-frame self-attention mechanisms improve character consistency but add significant computational complexity, limiting frame count and introducing redundancy, which affects temporal consistency and quality. To address these issues, we propose an adaptive motion-guided cross-frame attention that reduces complexity while preserving semantic details and consistency. By selectively incorporating moving regions based on optical flow sampling, we enable editing more frames jointly without extra computational cost. For longer videos, we introduce KV-caching of edited frames to enhance intermediate frame quality and temporal consistency, allowing our approach, Ada-VE, to achieve 3x more keyframes and up to 4x speed-up across 40 frames without quality loss.

Audio separation in real-world scenarios, where mixtures contain a variable number of sources, presents significant challenges due to limitations of existing models, such as over-separation, under-separation, and dependence on predefined training sources. We propose OpenSep, a novel framework that leverages large language models (LLMs) for automated audio separation, eliminating the need for manual intervention and overcoming source limitations. OpenSep uses textual inversion to generate captions from audio mixtures with off-the-shelf audio captioning models, effectively parsing the sound sources present. It then employs few-shot LLM prompting to extract detailed audio properties of each parsed source, facilitating separation in unseen mixtures. Additionally, we introduce a multi-level extension of the mix-and-separate training framework to enhance modality alignment by separating single source sounds and mixtures simultaneously. Extensive experiments demonstrate OpenSep's superiority in precisely separating new, unseen, and variable sources in challenging mixtures, outperforming SOTA baseline methods. 

As deep neural networks evolve from ConvNets to advanced vision transformers (ViTs), eliminating redundant data for faster processing without losing accuracy becomes crucial. Current methods are often architecture-specific or require re-training, limiting their flexibility. We address this by revealing a new property of lightweight ConvNets: their capacity to identify key discriminative patches in images independently of model accuracy or size. We show that suppressing fully-connected layers through simple weight recalibration improves patch localization. Building on this, we introduce PaPr, a patch pruning method for reducing redundancy across various architectures—ConvNets, ViTs, and hybrids—without re-training. PaPr achieves up to 70% patch reduction in videos with less than 0.8% accuracy loss and up to 3.7x FLOPs reduction, outperforming existing methods by 15% in reduction with 2.5% higher accuracy.

Visual sound source localization is challenging, especially in identifying each sound source's region in multi-source videos. Current self- and weakly supervised methods struggle in complex multi-source scenarios due to limited audio-visual correspondence. To address this, we introduce T-VSL, which incorporates text as an intermediate guide using tri-modal embeddings (e.g., AudioCLIP) to disentangle audio-visual correspondence. T-VSL predicts sound classes and uses text representations to refine source localization, allowing flexible source handling and strong zero-shot transfer to unseen classes. Experiments on MUSIC, VGGSound, and VGGSound-Instruments show significant improvements over current methods.

Separating single-source sounds in multi-source audio mixtures without single-source training data is a long-standing challenge. Existing methods struggle with multi-source mixtures due to the lack of single-source supervision. However, in language-conditional audio separation, we have text descriptions for each mixture in the training data, which serve as rough representations of the audio. In this paper, we propose a bi-modal separation framework that enhances unsupervised separation by leveraging signals in the conditioning modality (language) to separate single-source audio without single-source supervision. Using a pretrained joint embedding model (CLAP), we show that our method improves unsupervised baselines by reducing the distribution shift between training and test data, achieving a 71% SDR boost—reaching 97.5% of supervised performance. Additionally, we demonstrate a 17% improvement in supervised learning when augmented with our weakly-supervised framework, enabling a robust semi-supervised approach for audio separation.

Despite progress in semi-supervised image object detection, video object detection still faces challenges: (1) strong performance relies on annotated frames, (2) annotating multiple video frames is costly and often redundant, and (3) current techniques for static images don’t leverage video motion dynamics. We introduce SSVOD, an end-to-end semi-supervised video object detection framework that uses motion dynamics to make use of unlabeled frames with sparse annotations. By using flow-warped predictions for temporal consistency, we assemble robust pseudo-labels with cross-IoU and cross-divergence for bounding boxes and class labels. To balance confirmation bias and noise, we combine hard and soft pseudo-labels with confidence thresholds. SSVOD shows notable improvements over existing methods on ImageNet-VID, Epic-KITCHENS, and YouTube-VIS datasets.

We introduce AVE-CLIP to exploit AudioCLIP pre-trained on large-scale audio-image pairs for improving inter-modal feature correspondence on video AVEs. We propose a multi-window temporal transformer based fusion scheme that operates on different timescales of AVE frames to extract local and global variations of multi-modal features. We introduce a temporal feature refinement scheme to increase contrast with the background. Our method achieves state-of-the-art performance on the publicly available AVE dataset with 5.9% mean accuracy improvement which proves its superiority over existing approaches.

An audio-visual event (AVE) aligns visual and auditory signals in a video segment, but precise AVE localization is challenging due to the need for multi-modal feature correspondence across short and long temporal interactions. Existing methods struggle with capturing these scales due to limited multi-modal training strategies. To address this, we introduce AVE-CLIP, a framework combining pre-trained AudioCLIP with a multi-window temporal transformer to handle different temporal scales. Our contributions: (1) a multi-stage training framework to integrate AudioCLIP with AVE localization via contrastive fine-tuning and multi-scale training, (2) a multi-domain attention mechanism for local-global feature fusion, and (3) a temporal refining scheme with event-guided attention and post-processing for handling background variation. AVE-CLIP achieves state-of-the-art performance on the AVE dataset with a 5.9% accuracy improvement.

We address two critical challenges with transfer learning via fine-tuning: (1) The required amount of fine-tuning greatly depends on the distribution shift from source to target dataset. (2) This distribution shift greatly varies by layer, thereby requiring layer-wise adjustments in fine-tuning to adapt to this distribution shift while preserving the pre-trained network’s feature representation. To overcome these challenges, we propose RL-Tune, a layer-wise fine-tuning framework for transfer learning which leverages reinforcement learning to adjust learning rates as a function of the target data shift. RL-Tune outperforms other state-of-the-art approaches on standard transfer learning benchmarks by a large margin, e.g., 6% mean accuracy improvement on CUB-200-2011 with 15% data.