Research Projects

Objectives

• Collaborated with committee members Professor Pingbo Tang (CMU), Professor Chris McComb (CMU), Professor Chris Hendrickson (CMU), and Professor Chris Miller (Akron University) to improve the reliability of infrastructure system inspection through human-machine cooperation.

Results

• Proposed a digital twin system for bridge inspection with automatic logging by Unity, Revit, and Ansys that can record time-series human-computer interaction data during simulated bridge inspections.

• Mining time-series human cognitive histories and workflows (timestamps, mouse clicks, movements, eye movements, user input text, geometrical information) to understand inspection strategies and find reusable patterns with causal discovery and inference.

• Discovered inspection strategies from inspectors' behavioral data and quantified the reliability of defects localized by multiple inspectors through process mining and knowledge graph analysis.

• Advanced large language models (GPT 4 has a broad knowledge of engineering) to build the inspection assistants to understand the user behaviors data and make recommendations (achieving 91.2% precision)

Objectives

• Collaborated with Professor. 

Results

• Designed a simulation environment of tower crane control by Unity and collected operation videos and histories of tower crane operators.

• Developed fatigue detection algorithm (MTCNN, MobileNet, and LSTM), trained on the multiple public fatigue datasets, and tested on the collected construction machinery operator’s data, achieving 92.81% detection accuracy.

• Analyzed variations in public human fatigue datasets such as real or pretend facial expressions, segment level of human-verified labeling, camera positions, acquisition scenarios, and illumination conditions.

• Gave guidance for building up a large and public realistic fatigue dataset for construction machinery operators.

Objectives

• Collaborated with three industrial partners: Fontusblue (Dr. Chris Miller), American Water (Dr. Ran Liu) and Ethos Collaboration (Damon Weiss, P.E.) and Professor Pingbo Tang (CMU) for the development of a water digital twin for maintenance-aware operation.

Results

•  Proposed a proactive digital twin water system with Unity and MATLAB Simulink simulation models to simulate anomaly situations and a Virtual Reality (VR) interface to capture the water system's human-in-the-loop inspection and operation process.

• Captured the operators' behaviors and vision trajectories and conducted pattern mining of visual trajectories for knowledge discovery and sharing between different operators through Augmented Reality (AR).

• Compared anomaly inspection performances between computer algorithms and human operators. Cooperation between operators and machines could improve anomaly detection recall.

•  Experiments demonstrate that for the simulated anomaly inspection and detection, the computer could achieve 0.712 recall, while operators could achieve 0.851 recall; combining computers and operators can achieve higher classification recalls (0.885).

•  The cooperative approach between computers and human operators has shown the potential to achieve higher anomaly classification accuracy, fostering a more robust, reliable system.

Objectives

• Collaborated with Professor Shuna Ni (University of Maryland, College Park), Professor Stoliarov, Stanislav I. (University of Maryland, College Park), and Professor Pingbo Tang (CMU) for development of data driven tools for fire pattern analysis.

Results

• Proposed an automatic indoor point cloud segmentation with Mask 3D and Segment Anything 3D for identifying the indoor objects (walls, ceilings, floors, windows, furniture) from indoor after-fire scenes. 

• Integrated Chatgpt4 and Midjourney (stable diffusion) to generate the different fire patterns image for data argumentation.

• Fine-tuned the segment anything model (SAM) with experts labeling for human-computer interaction in fire shapes segmentation and classification from indoor after-fire scenes with 95% accuracy.

• Designed a web app to help the fire investigators through Streamlit with the fine-tuned segment anything model.

• Traced the fire patterns back to 3D space to understand the spatial relationships with knowledge graphs to help the fire investigators identify the fire origins.

Objectives

• Collaborated with Professor. Alper Yilmaz (The Ohio State University), Dr. Ron Boring (Idaho National Laboratory), Professor George Edward Gibson Jr. (Arizona State University) for “Intelligent Process Visualization through Nuclear Operation Process Modeling, Reasoning, and Object Detection from Field Videos”

Results

• Aimed to increase personnel safety and reduce operating costs in Nuclear Power Plants (NPPs) by integrating computer vision and process reasoning methods for proactive safety visualization.

•  Conducted literature reviews, surveys, and interviews to synthesize NPP operational knowledge models.

• Three categories of information were identified: workspace dynamics, workflow prognostics, and hazards.

•  Developed natural language processing (BERT) for generating and updating fieldwork process models.

• Proposed a lifelong object detection framework for continuous learning from dynamic contexts.

• Developed an Extended Reality (XR)-augmented visual assistance framework, integrating Virtual Reality (VR) and Augmented Reality (AR) to capture and transfer inspection strategies for efficient operation guidance.

• Achieved an accuracy of 0.883 in sensor log analysis for predicting the next control action.

• Extracted over 80% of critical information from paper-based procedures using NLP.

• Demonstrated an average accuracy of 95.3% in object detection using computer vision algorithms.

Objectives

• Collaborated with Professor. Alper Yilmaz (The Ohio State University), Dr. Ron Boring (Idaho National Laboratory), Professor George Edward Gibson Jr. (Arizona State University) for “Intelligent Process Visualization through Nuclear Operation Process Modeling, Reasoning, and Object Detection from Field Videos”

Results

• Secondary development on commercial robot Misty to collect human response through touch, bump sensors or interactive communication

• Quantified influences of social robots on collecting IEQ responses compared to the traditional methods

Objectives

• Addressed the inefficiencies in the design process of Modular High-Rise Residential Buildings (MHRBs) traditionally reliant on designer experience

•  Addressed the challenges of time consumption and potential errors in the manual creation and adjustment of structural models, specifically focusing on shear wall structures.

Results

• Addressed the inefficiencies in the design process of Modular High-Rise Residential Buildings (MHRBs) traditionally reliant on designer experience.

• Developed a two-stage Genetic Algorithm (GA) to automate the layout design process for MHRBs.

•  Utilized Autodesk CAD's application programming interface to generate Computer-Aided Design (CAD) drawings automatically based on the GA framework.

• Provided two real-world examples to demonstrate the efficacy of the automated layout design approach 

• Developed a semi-automatic generation method to streamline the creation of structural models with minimal manual operation.

•  Utilized graph structure and depth-first search for quick room identification.

• Included a complete process for generating structural members like shear walls, beams, and slabs.

•  Achieved time consumption generally within one minute for the entire modeling process.

Objectives

• Collaborated with Professor. Jiepeng Liu

Results

• Model the design clash detection and resolution problem as the path-planning process.

• Introduced multi-agent reinforcement learning (MARL) system integrated with BIM by Deep Q-Network for automatic clash-free rebar designs in RC frames and precast concrete exterior walls.

• Utilized Generative Adversarial Network (Pix2pix) to learn from existing design drawings and generate preliminary 2D rebar designs.

• Demonstrated a reduction in engineering time for rebar designs by up to 80% compared to manual methods, while still adhering to design codes and achieving clash-free designs.

Objectives

• Collaborated with Professor. Jiepeng Liu

Results

• Aimed to improve the construction quality, efficiency, and environmental sustainability by modernizing the inspection process of Precast Concrete Elements (PCEs).

• Proposed the use of Terrestrial Laser Scanners (TLSs) for batch inspection of PCEs to overcome the limitations of traditional manual methods.

• Developed a segmentation and recognition approach to automatically identify various types of PCEs from large sets of outdoor laser scan data.

• Employed image processing techniques combined with the Radially Bounded Nearest Neighbor Graph (RBNN) algorithm to speed up data segmentation.

• Utilized as-designed models of PCEs from Building Information Modeling (BIM) for both coarse and fine matching to recognize PCE types.

• Conducted experimental studies on 22 PCEs involving 12 different types, confirming the effectiveness and efficiency of the proposed method.