Teaching

Probability and Random Processes

Probability theory is a branch of mathematics that reached maturity in the early 20th century, and has since been applied to many fields including radar, wireless communications, networking, control, data analysis, quantum physics, finance, and of course, gambling. It provides a systematic way to handle uncertainty, whether due to natural physical phenomena such as thermal noise, or due to incomplete information e.g. in financial markets. Accurate probabilistic models have been at the core of cool inventions ranging from inter-planetary spacecraft to medical imaging of the human body to instant spelling auto-correction on smartphones. While the mathematics required to understand probability and random processes can be scary, I try hard to always illuminate the bigger picture so that students don't miss the woods for the trees. Students who put in the effort to learn the material in these basic courses will build a strong foundation for the study of engineering applications of probability.

Signals and Systems

This is a course that I taught at U of T a few times, with the famous Oppenheim and Willsky textbook. A signal is any measurable variation of a physical or mathematical object over time or space. This includes air pressure, electromagnetic field strength, temperature, altitude, number of members of a population, etc. In other words, signals are everywhere! In an undergraduate course on signals and systems, students are first introduced to the concept of a signal as a real function of a real variable, usually denoted x(t) or x[n], with the former representing a continuous-time (CT) signal in which the independent variable t is real-valued, while the latter represents a discrete-time (DT) signal in which n is an integer. Signals are then classified in various ways -- periodic versus aperiodic, odd/even symmetric versus asymmetric, finite energy versus finite power. Then we zoom in on periodic signals and show that it is possible to represent all practical periodic signals using a weighted sum of an infinite number of sinusoids at the harmonic frequencies. This beautiful result is due to Fourier, and is known as the Fourier series. Then we would let the period of a periodic signal grow to infinity, and arrive at the Fourier analysis of aperiodic signals, known as the Fourier transform. Now all signals we will encounter can be given a "frequency-domain" representation, which offers an alternative description of the same object. The frequency domain is like the flip side of a coin (the signal) to the time domain, and both give a complete description of the signal. However, some things are easier to understand in the frequency domain than the time domain and vice versa, and both domains are needed in signal analysis.

In both natural and engineered systems, signals are almost always passed through one or more input-output systems before they can be observed or used. For instance, to derive a trend in a stock index, we need to smooth out the very noisy short-term gyrations. We do this by passing the original stock index "signal" through a smoothing filter "system" and use the system's "output" to derive the information we want, i.e. the trend line. How do we design systems to accomplish a desired task such as smoothing; conversely how do we understand what a naturally-occurring system is doing to its input signal? In this course, we would first introduce the language for describing systems -- linear versus non-linear, memory-less versus with-memory, time-varying versus time-invariant. Then we describe the theory behind linear time-invariant (LTI) systems, which is both elegant and powerful, and allows us to talk about such useful concepts as amplifiers and filters.

Finally, we would talk about the link between CT and DT signals that sampling provides. Sampling at a constant rate is the basic operation performed at the front-end of every digital device that has to interface with CT signals, e.g. phones, cameras and sensors. Can a CT signal be completely represented using only samples taken at a finite rate? Surely some information must be lost if we just retain one value every T seconds! But a frequency-domain description quickly dispels this notion, and leads to the Nyquist-Shannon sampling theorem, one half of the theoretical bedrock upon which all digital systems are built. (The other half is Shannon's source coding theorem, which shows that we can not only discretize in time, but in amplitude too so that any practical message can be represented with only a finite number of bits, even if it started its life as a CT signal.) With a good appreciation of this basic theoretical analysis and design of signals and systems, the student will be ready to appreciate many important applications encountered daily.

Digital and Analog Communications

Analog communications refers to the transmission of a signal, usually an audio or a video signal, by reproducing at the receiver a replica of the continuous-time continuous-amplitude (or analog) signal generated at the transmitter. The difficulty is that audio and video signals cannot travel very far without processing (just think of how far your voice can travel or how difficult it is to signal visually to someone). This process is known as modulation, and requires the study of Fourier transforms. But the theory underpinning analog communications says that if a signal has a bandwidth of W Hz, then we must use at least that much bandwidth to transmit it, even in an ideal channel with no noise or interference. Therefore in a finite band, say the 88 to 108 MHz FM band, it is theoretically possible to only transmit a finite number of FM radio channels, even if the SNR is infinite. But if the SNR is higher than we need it to be (resulting in a signal quality that is better than needed), can we not reduce the bandwidth so that the quality if just what we need? In other words, can bandwidth be traded off with SNR? This question was rigorously answered in the 1940s.

The realization of the inadequacy of Fourier transforms as a foundation for information communication led Claude Shannon to formulate a new theory of communication, based on probabilistic reasoning. Shannon's theory became known as information theory, and models all communication channels as information pipes, with widths that constraint the rate at which information can flow through them. The width of the pipe is determined by the probabilistic model linking the transmitter to the receiver (technically the conditional channel output distribution given the input), and the maximum rate of information flow is known as the capacity of that channel. This new point of view turned some of the existing thinking on its head e.g. up to then, it was accepted that once there was noise (i.e. SNR is finite) then there was no way to transmit a signal with full integrity, i.e. to recover the transmitted signal perfectly. Information theory showed this to be untrue, and in fact that at any SNR, as long as the transmitted information rate was below the channel capacity, communication could in theory be achieved with full integrity. From Shannon's theoretical foundations, research into practical means to realize at least part of this theoretical potential took off, led primarily by AT&T Bell Labs, which has today led us to the Internet, LTE, Wi-Fi, digital TV, USB, and so many other communication links that operate at bit rates that Shannon and his contemporaries would not have dare to dream about.

In courses on digital and analog communications, the rich theory of modern communication systems will be discussed, building on the fundamental mathematical tools of signal and system theory, and probability theory.