Variational Bayes and a problem of reliable communication

This lecture is devoted to the study of a problem in reliable communication using a variational Bayes method. The variational method expresses a Bayesian posterior distribution as the unique minimizer of a quantity dubbed apparent information. This has the same nature as the free energy of statistical mechanics. The minimum apparent information coincides with the full information of the observation. Reliable communication over an error prone channel can be achieved by the use of random block coding, as originally proposed by Shannon. The primary Bayesian problem in this context, is that of estimating the transmitted block from observations of the output of the channel. Scaling limits for the various information quantities are derived for this problem; these show that the primary problem undergoes a second-order phase transition, in a very precise sense, at the channel capacity; the code rate is shown to play the role of absolute temperature. Shannon’s reliability function is recovered as the scaling limit of the full information of a secondary Bayesian problem, in which the channel noise and random code are estimated from the observation of a block decoding error. This secondary problem undergoes a third-order phase transition at a second critical code rate. 

Based on http://www.mit.edu/~mitter/publications/118_variational_bayes_I_CIS.pdf

Sanjoy K. Mitter received his Ph.D. degree from the Imperial College of Science and Technology, University of London, in 1965. He had previously worked as a research engineer at Brown Boveri & Co. Ltd., Switzerland (now ASEA Brown Boveri) and Battelle Institute in Geneva, Switzerland. He taught at Case Western Reserve University from 1965 to 1969 and joined MIT in 1969, first as a Visiting Professor and then in 1970 as Associate Professor in the Department of Electrical Engineering and Computer Science and in 1973 as a Professor of Electrical Engineering. He was the Director of the MIT Laboratory for Information and Decision Systems from 1981 to 1999 and Director of the Center for Intelligent Control Systems, an inter-university (Brown-Harvard-MIT) center for research on the foundations of intelligent systems from 1986-2000. He has held visiting positions at the Tata Institute of Fundamental Research, Bombay, India; Scuola Normale Superiore, Pisa, Italy; Imperial College of Science and Technology; Institut National de Recherche en Informatique et en Automatique, France; University of Groningen, the Netherlands; ETH, Zürich, Switzerland and several universities in the United States including the University of California, Berkeley where he was the McKay Professor in March 2000, and held the Russell-Severance-Springer Chair in Fall 2003.

Professor Mitter's research has spanned the broad areas of Systems, Communication and Control. Although his primary contributions have been on the theoretical foundations of the field, he has also contributed to significant engineering applications, notably in the control of interconnected power systems, character recognition, and automatic recognition and classification of electrocardiograms. His current research interests are theory of stochastic dynamical systems, nonlinear filtering, stochastic and adaptive control; mathematical physics and its relationship to system theory; image analysis and computer vision; and structure, function and organization of complex systems.

Professor Mitter has served on several advisory committees and editorial boards for IEEE, SIAM, AMS, NSF and ARO. He is currently Associate Editor of the Journal of Applied Mathematics and Optimization; Random and Computational Dynamics; Sankhya and the Ulam Quarterly and Editor-at-Large for Communications in Information and Systems. He is a fellow of the IEEE and was the recipient of the 2000 IEEE Control Systems Award. In 1988 he was elected to the National Academy of Engineering. He was elected a Foreign Member of Istituto Veneto di Scienze, Lettere ed Arti in 2003.