Multimedia Signal Processing

Introduction

Course Schedule and Reading

Resource

Bibliography

Introduction

Course Schedule and Reading

Introduction. Multimedia signal. Audio, Voice, Image, Video, Text. (Salomon and Motta, Cap. 1) (notebook image introduction 1, 2, 3)
Spectrogram (notebook)
Digital Signal Processing. Resampling. Multirate Signal Processing. Oversampling. (Oppenheim)
Scalar Quantization and Vector Quantization. (notebook) (Gray and Neuhoff, 1998) (Gray, 1984) (Cap. 2, 3 You, 2010)
Basic compression. RLE. Dictionary methods. Statistical methods. (Salomon and Motta, Cap. 2, 5, 6) (RLE, fileformat) (rle notebook) (move-to-front notebook) (slides)
Orthogonal transform. Discrete Cosine Transform (DCT). Wavelets. (Cupertino, 2002)
Image compression. JPEG Standard. (Wikipedia, ITU-T T.81), (Kerr, 2012)
Linear Prediction. Differential Pulse Code Modulation (DPCM). Linear Predictive Coding (LPC). (Yehia, 1993) (slides dpcm, lpc)
Audio and voice compression. (Gray, preprint) (wikipedia) (Cap. 1, 6, 7, 10 You, 2010)
Speech (vocal apparatus video 1, 2, 3)
Vision (khanacademy), Hearing (TED-ed, khanacademy), Perception (Kandel).
Video compression. MPEG Standard.
Principal Component Analysis (Shlens 2014) (slides pca)

Resource

Bibliography

Salomon, D., Data Compression: The Complete Reference, Springer. (google books, archive)
Salomon, D., Motta, G., Handbook of Data Compression, Springer. (google books)
Gibson, J. D., Multimedia Communications: Directions and Innovations, Academic Press. (google books)
Oppenheim, A.V., Schafer, R., Digital Signal Processing, Prentice-Hall. (google books)
Gonzalez, R.C., Woods, R.E., Digital Image Processing, Pearson. (google books)
Sayood, K., Introduction to Data Compression. Morgan Kaufmann. (google books)
Bocharova, I., Compression for Multimedia. Cambridge University Press. (google books)
Huang, X., Acero, A., Hon, H., Spoken Language Processing. Prentice Hall. (google books)

Suggested Readings

Gallager, R.G., Principles of Digital Communications, Chapter 3. (MIT)
Gray, R., Neuhoff, D. L., Quantization, IEEE Transactions on Information Theory, 44(6), October 1998. (IEEE)

Abstract:The history of the theory and practice of quantization dates to 1948, although similar ideas had appeared in the literature as long ago as 1898. The fundamental role of quantization in modulation and analog-to-digital conversion was first recognized during the early development of pulse-code modulation systems, especially in the 1948 paper of Oliver,Pierce, and Shannon. Also in 1948, Bennett published the first high-resolution analysis of quantization and an exact analysis of quantization noise for Gaussian processes, and Shannon published the beginnings of rate distortion theory, which would provide a theory for quantization as analog-to-digital conversion and as data compression. Beginning with these three papers of fifty years ago, we trace the history of quantization from its origins through this decade, and we survey the fundamentals of the theory and many of the popular and promising techniques for quantization.

Gray, R. Vector quantization. IEEE ASSP Magazine, 1 (2), 4-29,1984. (IEEE)

Abstract: A vector quantizer is a system for mapping a sequence of continuous or discrete vectors into a digital sequence suitable for communication over or storage in a digital channel. The goal of such a system is data compression: to reduce the bit rate so as to minimize communication channel capacity or digital storage memory requirements while maintaining the necessary fidelity of the data. The mapping for each vector may or may not have memory in the sense of depending on past actions of the coder, just as in well established scalar techniques such as PCM, which has no memory, and predictive quantization, which does. Even though information theory implies that one can always obtain better performance by coding vectors instead of scalars, scalar quantizers have remained by far the most common data compression system because of their simplicity and good performance when the communication rate is sufficiently large. In addition, relatively few design techniques have existed for vector quantizers. During the past few years several design algorithms have been developed for a variety of vector quantizers and the performance of these codes has been studied for speech waveforms, speech linear predictive parameter vectors, images, and several simulated random processes. It is the purpose of this article to survey some of these design techniques and their applications.

Graham Hudson et al, JPEG at 25: Still Going Strong, 24, 2, 2017 (IEEE).
Vaidyanathan, P.P., The Theory of Linear Prediction, Morgan & Claypool Publishers, 2008. (google books, archive)
Yehia, H. C., Análise de funções de erro em sistemas de codificação LPC, ITA, 1993. (ITA) - capítulos 1 e 2
John Makhoul, Linear Prediction: A Tutorial Review, Volume: 63, Issue: 4 , April 1975 (IEEE)
Atal, B. S., The History of Linear Prediction, IEEE Signal Processing Magazine 23 (2), 2006. (IEEE)

Abstract: This paper recollects the events that led to proposing the linear prediction coding (LPC) method, then the multipulse LPC and the code-excited LPC.

Lima, P. C., Wavelets: uma introdução, Matemática Universitária, 33, 2002. (matemática universitária)
Gomes, J., Velho, L., From Fourier Analysis to Wavelets, IMPA Springer, 2015. (google books)
Gomes, J., Velho, L., Goldenstein, S., Wavelets: Teoria, Software e Aplicações, 21 Colóquio Brasileiro de Matemática, 1997.
Schroeder, M., Atal , B., Code-excited linear prediction(CELP): High-quality speech at very low bit rates, ICASSP, 1985. (IEEE)

Abstract: We describe in this paper a code-excited linear predictive coder in which the optimum innovation sequence is selected from a code book of stored sequences to optimize a given fidelity criterion. Each sample of the innovation sequence is filtered sequentially through two time-varying linear recursive filters, one with a long-delay (related to pitch period) predictor in the feedback loop and the other with a short-delay predictor (related to spectral envelope) in the feedback loop. We code speech, sampled at 8 kHz, in blocks of 5-msec duration. Each block consisting of 40 samples is produced from one of 1024 possible innovation sequences. The bit rate for the innovation sequence is thus 1/4 bit per sample. We compare in this paper several different random and deterministic code books for their effectiveness in providing the optimum innovation sequence in each block. Our results indicate that a random code book has a slight speech quality advantage at low bit rates. Examples of speech produced by the above method will be played at the conference.

Rabiner, L. R., Schafer, R. W., Digital Processing of Speech Signals, Pearson, 1978.
Gray, R. M., Linear Predictive Coding and the Internet Protocol: A survey of LPC and a History of of Real time Digital Speech on Packet Networks, Journal Foundations and Trends in Signal Processing, 3 (3), 2010. (preprint)
Musmann, H. G., Genesis of the MP3 audio coding standard, IEEE Transactions on Consumer Electronics, 52 (3), 2006. (IEEE)
Karwowski, D., et al, 20 Years of Progress in Video Compression - from MPEG-1 to MPEG-H HEVC, International Conference on Image Processing and Communications, 2017.

Abstract: Compression of moving images has opened unprecedented opportunities of transmission and storage of digital video. Extraordinary performance of today’s video codecs is a result of tens of years of work on the development of methods of data encoding. This paper is an attempt to show this history of development. It highlights the history of individual algorithms of data encoding as well as the evolution of video compression technologies as a whole. With the development of successive technologies also functionalities of codecs were evolving, which make also the topic of the paper. The paper ends the attempt of authors' forecasting about the future evolution of video compression technologies.

You, Y., Audio Coding: Theory and Applications, Springer, 2010. (google books)
Kerr, D. A., Chrominance Subsampling in Digital Images, 2012. (online)
Encyclopedia of Graphics File Formats: https://www.fileformat.info/mirror/egff/index.htm
Dave Litwiller, CCD vs. CMOS: facts and fiction, Photonics spectra 35(1):154-158 · January 2001. (online)
Richard W. Harold, An introduction to apperance analysis, SS Number 84, 2001 (link)
Karlheinz Brandenburg, MP3 and AAC explained, 1999 (link)
J. Zeng, O. C. Au, W. Dai, Y. kong, L. Jia, W. Zhu, A Tutorial on Image/Video Coding Standards, APSIPA, 2013 (link)
Russ, J. C., The Image Processing Handbook, CRC Press, 2006 (google books)
Lepton image compression (link), Guetzli (link), WaveNet (link)
TED talks: How computers learn to recognize objects instantly (link); How we teach computers to understand pictures (link)
Computer-based synthesis of speech and song on IBM 704 (link) (yt-link)

Notebooks and examples

Introdução (nbviewer) (colab)
imagem em cores (nbviewer) (colab)
Normalização de histograma (nbviewer) (colab)
move-to-front (nbviewer) (colab)
RLE (nbviewer) (colab)
Quantização escalar (nbviewer) (pdf) (colab)
Dithering, Floyd-Steinberg (nbviewer) (colab)
Reamostragem (nbviewer) (colab)
Filtros polifásicos (nbviewer) (colab)
Espectrograma e Tom Shepard (nbviewer) (colab)
Modelo AR (nbviewer) (colab)
DPCM (nbviewer) (colab)
DCT (nbviewer) (colab)
k-médias (nbviewer) (colab)
Vogais LPC (nbviewer) (colab)
Síntese LPC (nbviewer) (colab)
JPEG (nbviewer) (colab)
PCA (nbviewer) (colab)
LPC Matlab, Mathworks
cjpeg - compress an image file to a JPEG file
Wavelets (nbviewer) (colab)

Google Sites

Report abuse