Research

The Memorial Sloan Kettering high-performance cell imaging project is accelerating blood and bone marrow cell identification to assist pathologists in real-time. Manual classification of cell morphologies in sample images is time-intensive. This web application improves the process through automated AI-powered detection mapped to an interactive cell atlas. To scale classification throughput, we are tiling images into segments for parallel processing. The backend is being rearchitected onto a serverless AWS architecture with Lambda functions. These functions enable low-latency concurrent tile classification, slashing processing time from minutes to seconds per image. Additional Lambdas are running deep learning classifiers to identify cell types from visual cues in each tile. By boosting cell quantification performance, this application can scale cancer diagnostic assistance across Memorial Sloan Kettering now. Pathologists are gaining rapid access to cell counts and types, empowering accelerated identification of hematologic diseases like leukemia.

The next frame prediction project is an automated early warning system for self-driving vehicles. It utilizes a convolutional LSTM neural network to ingest sequential dashcam footage and forecast the next likely road frame. By comparing real-time inputs to learned driving patterns, deviations can be rapidly detected. For example, sudden braking or swerving events are flagged as abnormalities before they transpire. This not only allows the prediction model to improve by learning from new edge cases, but more importantly enables preemptive reactions. Built on scraped video data of regular traffic flow, the system has learned to expect certain continuations. When the actual sequence differs, the car can adjust ahead of time rather than responding last-minute. This propensity for looking ahead promises to enhance autonomous navigation, preempting accidents by alerting vehicles to irregularities in typical road conditions before they occur.

The CLIP Multimodal project teaches computers to bridge understanding across multiple data types like text, images, and audio. Traditional machine learning manipulates one data type, while newer models can classify input modalities—for example, predicting text sentiment. We aim higher: true multimodal comprehension, allowing conversions between modalities. This requires innovating analysis methods for each type. We extend CLIP, which relates images and text through contrastive learning. Now it will ingest three data streams—visual, textual, auditory—gleaning connections across them. For example, it could generate captions for photos or select suitable background music given a text prompt. Building computers that link concepts across modes, regardless of input/output type, is essential as multimodal inputs and applications proliferate.

Stratus is a cloud API that makes it easier for anyone to train and deploy task-specific machine learning in a matter of seconds. It is meant as a give-and-go solution for discrete prediction tasks, enabling people with or without a background in ML to start harnessing the technology. All you need to provide is the data, Stratus does the rest. 

We are also working on a more robust implementation of Stratus, enabling super large model training and deployment, on the scale of billions of parameters.

This project explored how information on the internet is stored by scraping data from diverse sources like YouTube, Facebook, and news outlets. We then created a knowledge graph - an interlinked collection of concepts, entities, relationships, and events. Knowledge graphs allow information storage and retrieval with huge potential. Digital assistants like Siri can't access our personalized data, but knowledge graphs analyzing information from our emails, messages, and apps could store salient details on-device. Then Siri could access this data to answer personalized questions without sending anything off-device or retaining machine learning models. Through rigorous web scraping, we figured out existing knowledge structures on the internet. We captured and preserved this information in an ever-growing knowledge graph using efficient methods. We also leveraged the vast amounts of collected data to build models for various downstream applications based on members' interests.