Research

ATICVis: A Visual Analytics System for Asymmetric Transformer Models Interpretation and Comparison (2023)

In recent years, natural language processing (NLP) technology has made great progress. Models based on transformers have performed well in various natural language processing problems. However, a natural language task can be carried out by multiple different models with slightly different architectures, such as different numbers of layers and attention heads. In addition to quantitative indicators such as the basis for selecting models, many users also consider the language understanding ability of the model and the computing resources it requires. However, comparing and deeply analyzing two transformer-based models with different numbers of layers and attention heads are not easy because it lacks the inherent one-to-one match between models, so comparing models with different architectures is a crucial and challenging task when users train, select, or improve models for their NLP tasks. In this paper, we develop a visual analysis system to help machine learning experts deeply interpret and compare the pros and cons of asymmetric transformer-based models when the models are applied to a user’s target NLP task. We propose metrics to evaluate the similarity between layers or attention heads to help users to identify valuable layers and attention head combinations to compare. Our visual tool provides an interactive overview-to-detail framework for users to explore when and why models behave differently. In the use cases, users use our visual tool to find out and explain why a large model does not significantly outperform a small model and understand the linguistic features captured by layers and attention heads. The use cases and user feedback show that our tool can help people gain insight and facilitate model comparison tasks.

[Paper in MDPI Applied Science]

DLA-VPS: Deep Learning Assisted Visual Parameter Space Analysis of Cosmological Simulations (2022)

Cosmologists often build a mathematics simulation model to study the observed universe. However, running a high-fidelity simulation is time consuming and thus can inconvenience the analysis. This is especially so when the analysis involves trying out a large number of simulation input parameter configurations. Therefore, selecting an input parameter configuration that can meet the needs of an analysis task has become an important part of the analysis process. In this work, we propose an interactive visual system that efficiently helps users understand the parameter space related to their cosmological data. Our system utilizes a GAN-based surrogate model to reconstruct the simulation outputs without running the expensive simulation. We also extract information learned by the deep neural-network-based surrogate models to facilitate the parameter space exploration. We demonstrate the effectiveness of our system via multiple case studies. These case study results demonstrate valuable simulation input parameter configuration and subregion analyses.

[Paper in IEEE Computer Graphics and Applications]

CNERVis: A Visual Diagnosis Tool for Chinese Named Entity Recognition (2021)

Named entity recognition (NER) is a crucial initial task that identifies both spans and types of named entities to extract the specific information, such as organization, person, location, and time. Nowadays, the NER task achieves state-of-the-art performance by deep learning approaches for capturing contextual features. However, the complex structures of deep learning make a black-box problem and limit researchers’ ability to improve it. Unlike the Latin alphabet, Chinese (or other languages such as Korean and Japanese) do not have an explicit word boundary. Therefore, some preliminary works, such as word segmentation (WS) and part-of-speech tagging (POS), are needed before the Chinese NER task. The correctness of preliminary works importantly influences the final NER prediction. Thus, investigating the model behavior of the Chinese NER task becomes more complicated and challenging. In this paper, we present CNERVis, a visual analysis tool that allows users to interactively inspect the WS-POS-NER pipeline and understand how and why a NER prediction is made. Also, CNERVis allows users to load the numerous testing data and explores the critical instances to facilitate the analysis from large datasets. Our tool’s usability and effectiveness are demonstrated through case studies.

[Paper in Journal of Visualization]

Efficient and Portable Distribution Modeling for Large-Scale Scientific Data Processing with Data-Parallel Primitives (2021)

The use of distribution-based data representation to handle large-scale scientific datasets is a promising approach. Distribution-based approaches often transform a scientific dataset into many distributions, each of which is calculated from a small number of samples. Most of the proposed parallel algorithms focus on modeling single distributions from many input samples efficiently, but these may not fit the large-scale scientific data processing scenario because they cannot utilize computing resources effectively. Histograms and the Gaussian Mixture Model (GMM) are the most popular distribution representations used to model scientific datasets. Therefore, we propose the use of multi-set histogram and GMM modeling algorithms for the scenario of large-scale scientific data processing. Our algorithms are developed by data-parallel primitives to achieve portability across different hardware architectures. We evaluate the performance of the proposed algorithms in detail and demonstrate use cases for scientific data processing.

[Paper in MDPI Algorithms]

InSituNet: Deep Image Synthesis for Parameter Space Exploration of Ensemble Simulations (2019)

IEEE VIS 2019 (SciVis) Best Paper Award

We propose InSituNet, a deep learning based surrogate model to support parameter space exploration for ensemble simulations that are visualized in situ. In situ visualization, generating visualization at simulation time, is becoming prevalent in handling large-scale ensemble simulations because of the I/O and storage constraints. However, in situ visualization approaches limit the flexibility of post-hoc exploration because the raw simulation data are no longer available. Although multiple image-based approaches have been proposed to mitigate this limitation, those approaches lack the ability to explore the parameters of ensemble simulations. Our approach allows flexible exploration of parameter space for large-scale ensemble simulations by taking advantage of the recent advances in deep learning. Specifically, we design InSituNet as a convolutional regression model to learn the end-to-end mapping from the simulation and visualization parameters to the visualization results. With the trained model, users can generate new images for different simulation parameters under various visualization settings, which enables in-depth analysis of the underlying ensemble simulations. We demonstrate the effectiveness of InSituNet in combustion, cosmology, and ocean simulations through quantitative and qualitative evaluations.

Project Advisor: Prof. Han-Wei Shen (The Ohio State University)

[Paper in IEEE Transactions on Visualization and Computer Graphics]

NNVA: Neural Network Assisted Visual Analysis of Yeast Cell Polarization Simulation (2019)

IEEE VIS 2019 (VAST) Best Paper Honorable Mention

Complex computational models are often designed to simulate real-world physical phenomena in many scientific disciplines. However, these simulation models tend to be computationally very expensive and involve a large number of simulation input parameters which need to be analyzed and properly calibrated before the models can be applied for real scientific studies. We propose a visual analysis system to facilitate interactive exploratory analysis of high-dimensional input parameter space for a complex yeast cell polarization simulation. The proposed system can assist the computational biologists, who designed the simulation model, to visually calibrate the input parameters by modifying the parameter values and immediately visualizing the predicted simulation outcome without having the need to run the original expensive simulation for every instance. Our proposed visual analysis system is driven by a trained neural network-based surrogate model as the backend analysis framework. Surrogate models are widely used in the field of simulation sciences to efficiently analyze computationally expensive simulation models. In this work, we demonstrate the advantage of using neural networks as surrogate models for visual analysis by incorporating some of the recent advances in the field of uncertainty quantification, interpretability and explainability of neural network-based models. We utilize the trained network to perform interactive parameter sensitivity analysis of the original simulation at multiple levels-of-detail as well as recommend optimal parameter configurations using the activation maximization framework of neural networks. We also facilitate detail analysis of the trained network to extract useful insights about the simulation model, learned by the network, during the training process. We perform two case studies and evaluate our results by comparing with the original simulation model outcomes as well as the findings from previous parameter analysis performed by our experts.

Project Advisor: Prof. Han-Wei Shen (The Ohio State University)

[Paper in IEEE Transactions on Visualization and Computer Graphics]

Ray-based Exploration of Large Time-varying Volume Data Using Per-ray Proxy Distributions (2019)

Abstract—The analysis and visualization of data created from simulations on modern supercomputers is a daunting challenge because the incredible compute power of modern supercomputers allow scientists to generate datasets with very high spatial and temporal resolutions. The limited bandwidth and capacity of networking and storage devices connecting supercomputers to analysis machines become the major bottleneck for data analysis such that simply moving the whole dataset from the supercomputer to a data analysis machine is infeasible. A common approach to visualize high temporal resolution simulation datasets under constrained I/O is to reduce the sampling rate in the temporal domain while preserving the original spatial resolution at the time steps. Data interpolation between the sampled time steps alone may not be a viable option since it may suffer from large errors, especially when using a lower sampling rate. We present a novel ray-based representation storing ray based histograms and depth information that recovers the evolution of volume data between sampled time steps. Our view-dependent proxy allows for a good trade off between compactly representing the time-varying data and leveraging temporal coherence within the data by utilizing interpolation between time steps, ray histograms, depth information, and codebooks. Our approach is able to provide fast rendering in the context of transfer function exploration to support visualization of feature evolution in time-varying data.

Project Advisor: Prof. Han-Wei Shen (The Ohio State University)

[Paper in IEEE Transactions on Visualization and Computer Graphics]

EDDA - Extreme-scale Distribution-based Data Analysis (2019)

The EDDA library aims at visualizing distribution data for uncertainty analysis.  The goal is to provide a unified data model with generic distribution representations for the development of uncertainty visualization algorithms.  The distribution models to support will be parametric distributions like Gaussian and GMM, un-parametric distributions like histogram and KDE, as well as joint distributions. These are encapsulated into C++ template classes.  Coupled with our experiences on developing regular and curvilinear-grid datasets in OSUFlow, we provide an API allowing to query for the distribution, or its random sample, of a given 3D position of the uncertain dataset.  The return of the query can be either the interpolated distribution or a Monte-Carlo sample of the distribution, depending on the need of the visualization algorithm.

Project Advisor: Prof. Han-Wei Shen (The Ohio State University)

[WebSite] [GitHub]

Statistical Super Resolution for Data Analysis and Visualization of Large Scale Cosmological Simulations (2019)

Cosmologists build simulations for the evolution of the universe using different initial parameters. By exploring the datasets from different simulation runs, cosmologists can understand the evolution of our universe and approach its initial conditions. A cosmological simulation nowadays can generate datasets on the order of petabytes. Moving datasets from the supercomputers to post data analysis machines is infeasible. We propose a novel approach called statistical super-resolution to tackle the big data problem for cosmological data analysis and visualization. It uses datasets from a few simulation runs to create a prior knowledge, which captures the relation between low- and high-resolution data. We apply in situ statistical down-sampling to datasets generated from simulation runs to minimize the requirements of I/O bandwidth and storage. High-resolution datasets are reconstructed from the statistical down-sampled data by using the prior knowledge for scientists to perform advanced data analysis and render high-quality visualizations. 

Project Advisor: Prof. Han-Wei Shen (The Ohio State University) and Jonathan Woodring (Los Alamos National Laboratory) 

[Paper in PacificVis 2019]

Image and Distribution Based Volume Rendering for Large Data Sets (2018)

Analyzing scientific datasets created from simulations on modern supercomputers is a daunting challenge due to the fast pace at which these datasets continue to grow. Low cost post analysis machines used by scientists to view and analyze these massive datasets are severely limited by their deficiencies in storage bandwidth, capacity, and computational power. Trying to simply move these datasets to these platforms is infeasible. Any approach to view and analyze these datasets on post analysis machines will have to effectively address the inevitable problem of data loss. Image based approaches are well suited for handling very large datasets on low cost platforms.  Three challenges with these approaches are how to effectively represent the original data with minimal data loss, analyze the data in regards to transfer function exploration, which is a key analysis tool, and quantify the error from data loss during analysis. We present a novel image based approach using distributions to preserve data integrity. At each view sample, view dependent data is summarized at each pixel with distributions to define a compact proxy for the original dataset. We present this representation along with how to manipulate and render large scale datasets on post analysis machines. We show that our approach is a good trade off between rendering quality and interactive speed and provides uncertainty quantification for the information that is lost.

Project Advisor: Prof. Han-Wei Shen (The Ohio State University)

[Paper in PacificVis 2018]

Statistical-based Package Development for VTK-m (2017)

VTK-m is a toolkit of scientific visualization for emerging processor architectures. VTK-m supports concurrency data analysis and visualization algorithm to drive large scale datasets by providing abstract models that can be applied algorithms to traditional CPU, TBB and GPU processor architectures. In Los Alamos National Laboratory, I develop statistical-based data analysis algorithm, such as multivariate entropy and histogram computation of datasets, for VTK-m. Also, a parallel multivariate Gaussian  Mixture Model (GMM) trainer is built based on VTK-m.

Project Advisor: Jonathan Woodring and Li-Ta Lo (Los Alamos National Laboratory)

[WebSite] [GitLab]

Statistical Visualization and Analysis of Large Data using a Value-based Spatial Distribution (2017)

The size of large-scale scientific datasets created from simulations and computed on modern supercomputers continues to grow at a fast pace. A daunting challenge is to analyze and visualize these intractable datasets on commodity hardware. A recent and promising area of research is to replace the dataset with a distribution based proxy representation that summarizes scalar information into a much reduced memory footprint. Proposed representations subdivide the dataset into local blocks, where each block holds important statistical information, such as a histogram. A key drawback is that a distribution representing the scalar values in a block lacks spatial information. This manifests itself as large errors in visualization algorithms. We present a novel statistically-based representation by augmenting the block-wise distribution based representation with location information, called a value-based spatial distribution. Information from both spatial and scalar spaces are combined using Bayes' rule to accurately estimate the data value at a given spatial location. The representation is compact using the Gaussian Mixture Model. We show that our approach is able to preserve important features in the data and alleviate uncertainty.

Project Advisor: Prof. Han-Wei Shen (The Ohio State University)

[Paper in PacificVis 2017]

Glyph-based Universe Visualization (2015)

(CSE5545: High Performance Visualization for Large-Scale Scientific Data Analytics)

Cosmological simulations produce massive data which is challenging to analyze and visualize. Identifying dark matter halos, which correspond to galaxies and clusters of galaxies, from a large number of particles is an active topic in cosmology. In this project, we propose a distribution-based superquadric glyph technique to visualize the velocity field of the halo. We use surface color texture to show the histogram of the vector field within a halo and use the glyph shape to show the principal components of the distribution as an overview of the velocity statistics. This glyph gives more details of the velocity direction distribution of the particles within a halo than a simple velocity dispersion. We explore various ways to visualize the merge tree. We construct a widget representing this tree, using it as a tool to select different time steps to view for the superquadric data, while locating and highlighting the corresponding selected halo (shown as a superquadric) in the scene. We also provide a merge tree visualization and glyph-based visualization system that allows users to navigate through the space interactively.

Course Advisor: Prof. Han-Wei Shen (The Ohio State University)

[Webpage]

Cloth Simulation (2011)

(CSE788: Physical Based Simulation)

In cloth simulation, the cloth is represented by many particles and springs and the particles are connected by the spring. There are three different kinds of force, stretch-compression force, shear force and bending force, in cloth simulation. The different spring types, structural spring, shear spring and bending spring, between the particles that can represent the different forces. The following figure shows the three kinds of the springs. The structure springs can handle the stretch and compression force and this spring connects every vertical and horizontal and neighboring particles. The shearing spring can handle the shearing force and this spring connects the diagonal particles. The bending spring can handle the bending force to represent the out-of-plane deformation and this spring connects the every other vertical and horizontal particles.

Course Advisor: Prof. Huamin Wang (The Ohio State University)

[Webpage]

Position Estimation Apparatuses and Systems and Position Estimation Methods Thereof  (2011)

A position estimation system is provided, including at least one measurement unit, a plurality of evaluation units and a particle filter. The at least one measurement unit obtains a first information, wherein the first information at least includes a motion information and a corresponding noise model of a traced object. Each of the evaluation units has a corresponding evaluation model, wherein each evaluation model generates a corresponding unit displacement estimation according to the first information. The particle filter samples and generates a plurality of displacement estimations according to the unit displacement estimations and the corresponding noise models respectively.

Project Advisor: Prof. Chieh-Chih Wang (National Taiwan University)

[Patent]

Monocular Simultaneous Localization, Mapping and Moving Object Tracking (2010)

Localization, mapping and moving object tracking serve as the basis for scene understanding. It has been demonstrated that LIDAR-based Simultaneous LocAlization, Mapping and Moving Object Tracking (SLAMMOT) can be accomplished in dynamic environments. In this paper, we propose an augmented state approach to accomplish more challenging monocular SLAMMOT in which both stationary and moving objects contribute to localization. It is demonstrated that augmented state SLAMMOT is superior to SLAM with DATMO in highly dynamic environments with approximately the same computational complexity. The proposed monocular SLAMMOT approach is accomplished using the extended Kalman filter (EKF) with the existing inverse depth parametrization. Moving object detection from a moving camera is accomplished by evaluating the local monocular SLAM results without adding new features and the results under the stationary feature assumption. The results from different frames are integrated using a binary Bayes filter for moving object detection. In addition, negative inverse depths are checked for detection which is integrated using a decision tree. Although monocular SLAMMOT inherits the observability issue of bearings-only tracking, we demonstrate that the proposed detection approach still detects moving objects reliably under unobservable conditions. The proposed monocular SLAMMOT approach reliably and effectively constructs 3D feature-based maps, detects moving objects and provides 3D pose and velocity estimates of the camera and moving objects using only visual images from a moving camera. The extensive simulations as well as experimental results demonstrate the feasibility of the proposed monocular SLAMMOT approach.

Project Advisor: Prof. Chieh-Chih Wang (National Taiwan University)

[Report]

Intelligent Phone and Home Robot，with Compal Communication Inc. (2009)

The intelligence and ability of robots grow at a fast pace in the past decades. Robots will go into everyone's family to serve in the near future. A fundamental requirement of a home robot is to receive the instruction from an authorized family member and move to correct location without hitting obstacles. However, each family is a new and unknown working environment for a home robot. The robot has to understand the house layout and learn family members' face quickly in the beginning. In this project, we develop simultaneously localization and mapping (SLAM) technique on a home robot prototype to build the unknown environment map. A face recognition technique only required small number of training images are deployed on the home robot. In addition, the robot can be remotely controlled by authorized user's hand posture. After taking the user's instruction, the robot executes the path planing and avoid obstacles on its path to the target location and complete the task. 

Project Advisor: Prof. Chieh-Chih Wang (National Taiwan University)

[Report]

Monocular Simultaneous Localization, Mapping and Moving Object Tracking (2010)

Body-language understanding is essential to human robot interaction, and hand posture recognition is one of the most important components in a body-language recognition system. The existing hand posture recognition approaches based on robust local features such as SIFT can be invariant to background noise and in-plane rotation. However the ignorance of the relationships among local features is a fundamental issue. The part-based models argue that objects of the same category share the same part-structure which consists of parts and relationships among parts. In this paper, a discriminative part-based model, Hidden Conditional Random Fields (HCRFs), is used to recognize hand postures. Although the existing global locations of features have been used to consider large scale dependency among parts in the HCRFs framework, the results are not invariant to in-plane rotation. New features by the distance to the image center are proposed to encode the global relationship as well as to perform in-plane rotation-invariant recognition. The experimental results demonstrate that the proposed approach is in-plane rotation-invariant and outperforms the approach using AdaBoost with SIFT.

Project Advisor: Prof. Chieh-Chih Wang (National Taiwan University)

[Paper in AIM 09]

HAND GESTURE RECOGNITION USING ADABOOST WITH SIFT (2007)

Hand posture understanding is essential to human robot interaction. The existing hand detection approaches using a Viola-Jones detector have two fundamental issues, the degraded performance due to background noise in training images and the in-plane rotation variant detection. In this paper, a hand posture recognition system using the discrete Adaboost learning algorithm with Lowe’s scale invariant feature transform (SIFT) features is proposed to tackle these issues simultaneously. In addition, we apply a sharing feature concept to increase the accuracy of multi-class hand posture recognition. The experimental results demonstrate that the proposed approach successfully recognizes three hand posture classes and can deal with the background noise issues. Our detector is in-plane rotation invariant, and achieves satisfactory multi-view hand detection.

Project Advisor: Prof. Chieh-Chih Wang (National Taiwan University)

[My Master Thesis] [Paper]

A VOD system on high-availability and load balancing Linux servers (2004)

In this paper, we integrate the technologies of high availability and load balancing clusters that combine features of both of these cluster types, increasing both the availability and scalability of services and resources. This type of cluster setup is commonly used for Web-based VOD servers.

Project Advisor: Prof. Chao-Tung Yang (Tunghai University)

[Paper in ICME 04] [Paper in PCM04]