Search this site

Home
Publications
Research
Open Positions
Contact

Home
Publications
Research
Open Positions
Contact
More

Contact

Fahad Shahbaz Khan

Professor and Deputy Department Chair of Computer Vision

MBZUAI

Masdar City,

Abu Dhabi, UAE

Email: fahad.khan@liu.se

fahad.khan@mbzuai.ac.ae

Large Multimodal Models for Detailed Visual Reasoning
Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this project, we aim to develop LMMs with more detailed visual reasoning capabilities. For instance, developing that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. Similarly, developing video conversational models with grounding capabilities or large multilingual multimodal models.

Efficient Visual Recognition on Edge and Mobile Devices
In recent years, transformer models have shown remarkable success in various vision applications such as classification, detection, and segmentation. However, deploying these models on resource-constrained mobile devices for real-time applications remains challenging due to their inherently complex. In this project, we aim to design new transformers architectures that are efficient yet accurate with the ability to seamlessly work on resource constrained devices such as, mobile phones.

Generative AI for Healthcare
Large language Models (LLMs) have the potential to impact the medical field for assisting diagnostic capabilities and efficiency. In this project, we aim to develop LLMs and LMMs for a variety of healthcare areas such as, generative model to understand and reason about chest Xrays, foundation models for pathology and ultrasound data, multilingual medical LLMs for different down-stream tasks, real-time breast lesion detection and universal medical image segmentation models.

Visual Object Tracking
Visual tracking is a classical computer vision problem with many applications. In generic tracking the task is to estimate the trajectory of a target in an image sequence, given only its initial location. In recent years, Siamese-based trackers have achieved promising performance in visual tracking. Most Siamese trackers employ hand-crafted modules to compute the similarity between the template and instances. Our research work focuses on investigating accurate and robust learnable modules to better capture semantic information from large-scale data during offline training.

Video Object Segmentation
The problem of video object segmentation (VOS) has a variety of important applications, including object boundary estimation for grasping, autonomous driving, surveillance and video editing. The task is to predict pixel-accurate masks of the region occupied by a specific target object, in every frame of a given video sequence. Our research work focuses on the semi-supervised setting, where a target ground truth mask is provided in the first frame. The main difficulty is to effectively handle appearance changes and similar background objects, while maintaining accurate segmentation. We strive to develop novel deep learning-based VOS models that can hangle large appearance changes, are fast, easily trainable and remains highly effective in cases of limited training data. .

Foundation Models for Remote Sensing
Remote sensing employs a wide range of sensor technologies to acquire data for earth observation and monitoring through satellites and aircrafts. The acquired data may differ in terms of Ground Sample Distance (GSD) based on sensor technology and altitude. GSD refers to the distance between the two adjacent pixels measured on ground. scale of the objects may vary considerably within a single image. In addition, multi-spectral (Sentinel-2) data captured from satellite utilize different sensors to acquire data. Consequently, multi-spectral data may possess different GSDs within the single image. Various bands combination in multi-spectral data can be used to highlight different information. This research strives to develop foundational models for remote sensing data that can be leveraged to solve various down-stream tasks.

Zero-shot Learning
Zero-shot learning strives to classify unseen categories for which no data is available during training. In the generalized variant, the test samples can further belong to seen or unseen categories. The stateof-the-art relies on Generative Adversarial Networks that synthesize unseen class features by leveraging class-specific semantic embeddings. During training, they generate semantically consistent features, but discard this constraint during feature synthesis and classification. We investigate how to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification. .

Human Action Recognition with Limited Supervision
Temporal action localization is a challenging computer vision problem. Here, the task is to find the start and end time (temporal boundaries or extent) of actions in a video. Most existing action localization approaches are based on strong supervision, requiring manually annotated ground-truth temporal boundaries of actions during training. However, framelevel action boundary annotations are expensive compared to video-level action label annotations. In this research, we strive to develop novel methods to localize actions in videos with limited supervision (leveraging only video-level supervision). .

Google Sites

Report abuse

Google Sites

Report abuse