Research Projects

at University of California, Davis

• In sequence decision making problems such as multi-armed bandit and reinforcement learning, there is an intrinsic tradeoff between exploration (of unknown environment) and exploitation (of current knowledge). Motivated by 1) seasonal sales variation in product recommendation, 2) load variation in network configuration, 3) price variation in agriculture treatments, note that: in a variety of applications for online decision making, the actual regret of making a suboptimal decision may vary depending on external conditions, leading to a various cost in exploration. Thus, there is an opportunity to explore in a relatively low cost and adaptively balance the exploration and exploitation tradeoff. This motivates us to focus on these new opportunistic learning problems.

In multi-armed bandit problem, for the scenario with an opportunistic learning cost, an AdaUCB algorithm has been proposed accordingly and its regret bound has been derived explicitly. The proposed algorithm out-performs classic UCB algorithms and can even achieve constant regret in scenarios with binary-distributed learning cost. This is an ongoing projects with more results coming soon.

• Paper published in ICML 2018.

at University of California, Davis

• In current practice, network configuration depends heavily on field experience and manual adjustment, which is labor-intensive, error-prone, and far from optimal. To address this issue, this project combines machine learning with network theory to automate and optimize network configuration.

1) An online-learning-based joint-optimization approach that combines neural network regression with Markov Chain Monte Carlo (MCMC) has been proposed for network utility maximization in cellular networks. Since the impact of a control parameter on network utility function is highly complex, a natural step is to learn it based on available network data. Further, since sequential decisions are needed in maximizing the overall network utility over a period of time, this problem falls into the general scope of reinforcement learning. However, general RL algorithms suffers from the extremely large state space and action set in this cellular network configuration problem. In contrast, we combine neural network regression with Gibbs-sampling theory to design an online framework that proved to converge to a local optimum promptly. Note that Gibbs-sampling is a widely used MCMC algorithm that fits the structure of cellular network in this problem.

In the algorithm design, several critical challenges, such as limited data availability and convoluted sample data, are addressed effectively by carefully designed techniques including data aggregation among different cells, and latent environment feature extraction. The proposed framework out-performs existing algorithms such as carefully designed multi-agent reinforcement learning baseline when tested on industrial-grade simulator.

Moreover, this work is extended to further combine transfer learning by noticing that cells in cellular networks are similar but not identical.

2) A learning-based task offloading framework has been proposed for vehicular network, which enables vehicles to learn the potential task offloading performance of its neighboring vehicles with excessive computing resources, and minimizes the average offloading delay. This work is based on multi-armed bandit (MAB) theory and opportunistic learning. The proposed algorithm converges fast to the optimal solution with theoretical performance guarantee, and numerically achieves close-to-optimal delay performance when simulated on a realistic highway scenario.

• Papers published in IEEE Big Data 2017, ICC 2018, GLOBECOM 2018, and IEEE IoT Journal.

at Tsinghua University

• To improve the delay performance of mobile cloud computing, edge cloud systems have been proposed. This research addresses the latency problem in edge cloud in a cost-efficiency way.

We have developed task assignment algorithms that jointly evaluate wireless and computational resources, and minimize both delay and servers’ costs in edge cloud systems. The delay performance has been further improved in a hybrid system with both edge and internet cloud. Also, optimal provisioning of the computational resource in the edge cloud is analyzed by Nash equilibrium. In this project, both mean delay and stringent delay requirements have been studied. To further consider vehicular cloud computing system, which is a various of edge cloud drawing increasing attentions, the optimal task replication policy for deadline violation probability has been designed.

• Papers published in ICC 2016, ICC 2017, and IEEE IoT Journal.

at Texas A&M University

• In scenarios of real-time networked control such as in-vehicular wireless network and other IoT systems, the conventional QoS such as delay is not proper to measure the performance of network transmission. To address this concern, we have proposed “inter-delivery time” as a new QoS metric in networked control scenarios, and designed optimal scheduling policies accordingly.

To capture the reliability requirement in networked control scenarios, we have formulated the problem as a risk-sensitive Markov decision process. For heterogeneous inter-delivery time requirements, the optimal policy and several computational efficient asymptotically optimal policies were designed. In obtaining the optimal policy for massive connectivity scenario, we have further extended the well-known Whittle's index policy into risk-sensitive case.

• Papers published in ICC 2015, MobiHoc 2015, INFOCOM 2015, and accepted by IEEE/ACM ToN Journal.

at Tsinghua University

• Base station (BS) sleeping is an effective way to improve the energy-efficiency of cellular networks. Yet, it may bring extra user-perceived delay. To study the fundamental energy-delay tradeoff in BS sleeping, in this research, we have formulated a theoretical framework and designed delay-constrained energy-optimal BS sleeping policies.

By queueing theory, this work conducted closed-form analysis to energy-efficiency and delay performance in BS sleeping scenarios. Several explicit energy-delay relationships were obtained. Surprisingly, we have found that the relationship is not always a tradeoff, i.e., there exist win-win regions for both energy and delay performance under derived conditions.

We have further designed energy-optimal BS sleeping policies with mean delay constraints. Practical constraints of BS sleeping such as setup time and detection cost during sleep are considered, and as a result a hysteresis sleep and several wake-up schemes are proposed. Furthermore, we found that, when delay constraint is smaller than a threshold, the optimal policy has a zero hysteresis time, and the optimal energy-delay relationship forms a linear tradeoff.

In addition, we have studied into the scenarios with bursty traffic by formulating the system as a partially observable Markov decision process. Research results show that the optimal sleeping policy is a two-threshold policy with wait-and-see nature. Also, we found that the traffic burstiness can enhance system performance on both energy and delay aspects.

This research has been published on IEEE ICCS 2012 (invited paper), ITC 2013 (best student paper award), GLOBECOM 2016, IEEE JSAC journal, IEEE Trans. Green Commun. & Netw., and SCIENTIA SINICA Informationis journal.

• Papers published in IEEE ITC, GLOBECOM, ICCS, JSAC journal, etc. Also, won the best student paper award in IEEE ITC 2013

at Tsinghua University

• Developed relay selection algorithms to optimize the energy-efficiency of wireless systems.

• Implemented on GNU Radio platform with USRP boards