Research

Research @ UPenn

Vision-based Navigation using Sensor Fusion

We study the perimeter defense game in a photo-realistic simulator and the real world, requiring defenders to estimate intruder states from vision. We train a deep machine learning-based system for intruder pose detection with domain randomization that aggregates multiple views to reduce state estimation errors and adapt the defensive strategy to account for this. We show that our approach improves state estimation, and eventually, perimeter defense performance in both 1-defender-vs-1-intruder games, and 2-defenders-vs-1-intruder games.


Graph Neural Networks for Imitation Learning

We leverage graph neural networks (GNNs) to develop an imitation learning framework that learns a mapping from defenders’ local perceptions and their communication graph to their actions. The proposed GNN-based learning network is trained by imitating a centralized expert algorithm such that the learned actions are close to that generated by the expert algorithm. Our GNN-based network is trained at a small scale and can be generalized to large-scale cases. We run perimeter defense games in scenarios with different team sizes and configurations to demonstrate the performance of the learned network.


Evaluation on Sym2Real Transfer

We aim to apply the theory derived from the perimeter defense problem to robots with realistic models of actuation and sensing and observe performance discrepancy in relaxing the first-order assumptions. The transition from theory to practice is detailed, and the designed system is simulated in Gazebo. Two metrics for parametric analysis and comparative study are proposed to evaluate the performance discrepancy.


Vehicle Localization using Sensor Fusion

The EKF-based multisensor fusion of Infrastructure node perception output from the camera and LiDAR with the vehicle’s low-cost inertial sensors (i.e., GPS and IMU) allows us to perform reliable vehicle localization without the need for relying on expensive inertial navigation systems or compute-intensive vision processing onboard the AVs. The proposed approach has been tested on real-world datasets collected from a test track in Ann Arbor, Michigan. 

 

Behavior Optimization using Game Theory

We study a variant of the pursuit-evasion game in the context of perimeter defense. In this problem, the intruder aims to reach the base plane of a hemisphere without being captured by the defender, while the defender tries to capture the intruder. We prove that the optimal strategies for both players are to move toward the optimal breaching point. Simulation results are presented to demonstrate that the optimality of the game is given as a Nash equilibrium.


Experimental Evaluation on Visual-SLAM

We propose the use of small-scale, flying robots that are able to localize themselves and autonomously navigate around obstacles. Because of the constraints on size, we rely on cameras which are the best low-power and lightweight sensors. We (a) analyze the effects of radioactive sources on camera sensors, affecting localization and mapping algorithms, (b) quantify these effects from a statistical viewpoint according to different source intensities, and (c) compare different solutions to mitigate these effects. 


SLOAM_RAL2020.mp4

SLAM with Semantic Lidar Odometry and Mapping

We propose a semantic feature-based pose optimization that simultaneously refines the tree models while estimating the robot pose. The pipeline utilizes a custom virtual reality tool for labeling 3D scans that are used to train a semantic segmentation network.



DARPA Subterranean Challenge

We participated in the first round of the Subterranean (SubT) Challenge hosted by the Defense Advanced Research Projects Agency (DARPA). We employed both legged and flying robots into underground environments to find artifacts, ranging from backpacks, cell phones, fire extinguishers, and dummy personnel, with a one-hour time limit. 


About the challenge:

Press:

Publication:

Tunnel Inspection using Semantic Image Segmentation

We propose MAVNet: a small, lightweight, deep neural network for real-time semantic segmentation on Micro Aerial Vehicles (MAVs). We use four classes to label pixels: background, corrosion, rivet, and water. We classify pixels to detect defects on concrete tunnel surfaces.


Research @ KAIST

Vehicle and Lane Detection with Occlusion Handling

In projects developing self-driving vehicles, we contribute to the robust perception system for detecting and localizing surrounding vehicles with lane information. For robust vehicle detection, v-disparity from stereo depth map is used for ground detection, and a deep convolutional neural network is employed to confirm occluded vehicles. The collected vehicle and lane information are tracked by Kalman filter, and the system shows higher localization accuracy after incorporating the knowledge of lane positions.


Robust Multi-Lane Detection and Tracking

Lane detection may suffer from challenging conditions such as occlusion, illumination changes, and poor road markings. To overcome these challenges, we apply an adaptive threshold to extract strong lane features and introduce an improved RANSAC algorithm using the feedback from the lane edge angles and the curvature of lane history.


Dual-mode Robot Delivering First Aid Kits

Search and Rescue (SAR) robot with dual mode (Aerial and Ground mode) delivers first aid kits to the survivor. The robot's audio and video transmission also relieve the victim.


Lateral Position Estimation of Surrounding Vehicles Using Stereo Vision

The feature-based lateral position estimation algorithm is proposed using stereo vision and provides lateral position regardless of viewpoint change and occlusion by extracting a pixel-wise feature. In the preprocessing step, v-disparity from the stereo depth map is calculated and used for ground detection. Then, vehicle candidates are created based on image thresholding and filtering, removing the ground portion from the camera image. These generated candidates are verified as vehicles by using a deep convolutional neural network. Inverse perspective mapping (IPM) is applied to the original image to estimate the lateral position of the grounded feature point.


Sensor Fusion for Indoor Drone Flying with Virtual Reality

We develop an indoor flying drone that can perform search and rescue missions in unknown environments. Since disaster sites are often dark and have limited satellite access, conventional Vicon-aided robot localization and camera-based object detection are not possible. To overcome these challenges, we propose a sensor fusion system that is robust to lighting constraints by utilizing lidar and stereoscopic IR. We employ virtual reality for rescuers to remotely access the disaster sites, which would prepare the drone to be ready for any unexpected scenario in tracking victims.


Research @ Cornell

SLAM for Autonomous Mobile Robot

I participated in the Autonomous Mobile Robot Competition held at Cornell University in 2015. The goal was for an iRobot Create to navigate through assigned waypoints, and competition milestones required novel algorithmic implementations. One of the major challenges was a cumulative position error incurred by errors in simultaneous localization and mapping (SLAM) due to slipping and noise as well as accidental bumps. I prevented the error by implementing a particle filter, which assigns high weights to the precisely localized particles when AR tags were found.

State Machine on Kuka Youbot

I built a Robot Operating System (ROS) based executive system on the KUKA youBot for a collaborative research project, namely Jarvis. I was responsible for the executive module where ensuring robust state transition is critical. To guarantee smooth transitions between seven different modules for the robot, e.g., perception, planner, control, etc., I designed the hierarchy of the system architecture and thus prevented undesired transitions by running a state machine using the SMACH library.