I am a Professor in the Information Processing Group at the School of Computer and Communication Sciences, EPFL. From 2003-2011, I was at the University of California, Berkeley, earning my tenure in 2008. My research interests are centered around Information Theory and Signal Processing. I am grateful to the generous support from the ERC Starting Grant "ComCom" (2011-2016) and from the Swiss National Science Foundation.
Attention with Markov: A Curious Case of Single-layer Transformers
Ashok Vardhan Makkuva, Marco Bondaschi, Alliot Nagle, Adway Girish, Hyeji Kim, Martin Jaggi, and Michael Gastpar. The Thirteenth International Conference on Learning Representation (ICLR 2025), Singapore, April 2025.
(Open Review version)
Transformers on Markov Data: Constant Depth Suffices
Nived Rajaraman, Marco Bondaschi, Kannan Ramchandran, Michael Gastpar, Ashok Vardhan Makkuva. (NeurIPS 2024), Vancouver, Canada, December 2024.
(Arxiv version)
Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
Adway Girish, Alliot Nagle, Marco Bondaschi, Michael Gastpar, Hyeji Kim, Ashok Vardhan Makkuva. (NeurIPS 2024), Vancouver, Canada, December 2024.
(Arxiv version)
Local to Global: Learning Dynamics and Effect of Initialization for Transformers
Ashok Vardhan Makkuva, Marco Bondaschi, Chanakya Ekbote, Adway Girish, Alliot Nagle, Hyeji Kim, Michael Gastpar. (NeurIPS 2024), Vancouver, Canada, December 2024.
(Arxiv version)
Variational Characterizations of Sibson's Alpha-Mutual Information
Amedeo Roberto Esposito, Michael Gastpar, Ibrahim Issa. IEEE International Symposium on Information Theory (ISIT 2024), Athens, Greece, July 2024. (Proceedings, arxiv extended version)
The Persuasion Bottleneck
Michael Gastpar and Aayush Rajesh. IEEE International Symposium on Information Theory (ISIT 2024), Athens, Greece, July 2024. (Proceedings)
Batch Universal Prediction
Marco Bondaschi and Michael Gastpar. IEEE International Symposium on Information Theory (ISIT 2024), Athens, Greece, July 2024. (Proceedings)
Properties of the Strong Data Processing Constant for Rényi Divergence
Lifu Jin, Amedeo Roberto Esposito and Michael Gastpar. IEEE International Symposium on Information Theory (ISIT 2024), Athens, Greece, July 2024. (Proceedings)
Simultaneous Computation and Communication over MAC
Matthias Frey, Igor Bjelakovic, Michael Gastpar, Jingge Zhu. IEEE International Symposium on Information Theory (ISIT 2024), Athens, Greece, July 2024. (Proceedings)
Lower Bounds on the Bayesian Risk via Information Measures
Amedeo Roberto Esposito, Adrien Vandenbroucque, and Michael Gastpar. Journal of Machine Learning Research (JMLR), 25(340):1-45, 2024.(Journal)
Impossibility results for estimation problems via f-divergences, Rényi divergence, and Maximal Leakage.
Alpha-NML Universal Predictors
Marco Bondaschi and Michael Gastpar. IEEE Transactions on Information Theory, 71(2):1171-1183, 2025. (Journal)
They interpolate between the Laplace estimator and the Normalized Maximum Likelihood estimator.
Shannon Bounds for Quadratic Rate-Distortion Problems
Michael Gastpar and Erixhen Sula. IEEE Journal on Selected Areas in Information Theory Special Issue in honor of Toby Berger, 5:597-608, 2024. (Journal, arxiv version)
Featuring the Gray-Wyner network and the CEO problem.
Generalization Error Bounds Via Rényi-, f-Divergences and Maximal Leakage
Amedeo Roberto Esposito, Michael Gastpar, and Ibrahim Issa. IEEE Transactions on Information Theory, 67(8):4986-5004, August 2021. (Journal, Arxiv version)
New classes of concentration bounds on the generalization error. For example, we show that the probability that the generalization error is larger than η is bounded by exp(-n η + L(D→A)), where the size of the training set D is n, the output of the learning algorithm is A, and L denotes Sibson's mutual information of order infinity.
Quantifying high-order interdependencies via multivariate extensions of the mutual information
Fernando E Rosas, Pedro AM Mediano, Michael Gastpar, Henrik J Jensen. Physical Review E, 100(3):032305, 2019. (Journal)
Proposal of a compact multivariate information measure, capturing synergy and redundancy.
Remote source coding under Gaussian noise: Dueling roles of power and entropy power
Krishnan Eswaran, Michael Gastpar. IEEE Transactions on Information Theory, 65(7):4486-4498, August 2019. (Journal, Arxiv version)
New converse bounds for the CEO problem with a general source observed in Gaussian noise.
The sampling rate-distortion tradeoff for sparsity pattern recovery in compressed sensing
Galen Reeves, Michael Gastpar. IEEE Transactions on Information Theory, 58(5):3065-3092 May 2012. (Journal)
Information theory of compressed sensing.
Compute-and-forward: Harnessing interference through structured codes
Bobak Nazer, Michael Gastpar. IEEE Transactions on Information Theory, 57(10):6463-6486, October 2011. (Journal)
New ways in cooperative communications.
Uncoded transmission is exactly optimal for a simple Gaussian "sensor" network
Michael Gastpar. IEEE Transactions on Information Theory, 54(11):5247-5251, November 2008. (Journal)
An intricate network for which a full and exact analysis can be given.
Computation over multiple-access channels
Bobak Nazer, Michael Gastpar. IEEE Transactions on Information Theory, 53(10):3498-3516, October 2007. (Journal)
Computation and Communication are two sides of one coin.
Cooperative strategies and capacity theorems for relay networks
Gerhard Kramer, Michael Gastpar, Piyush Gupta. IEEE Transactions on Information Theory, 51(9):3037-3063, September 2005. (Journal)
Information-theoretic principles for networks with many relaying nodes.
To code, or not to code: Lossy source-channel communication revisited
Michael Gastpar, Bixio Rimoldi, Martin Vetterli. IEEE Transactions on Information Theory, 49(5):1147-1158, May 2003. (Journal)
All instances where uncoded transmission is as good as the best error-control code.
Since 2011: Professor, EPFL
2003-11: Assistant, then Associate (w/tenure) Professor, University of California, Berkeley
1999-2002: PhD thesis, EPFL. 1997-1998: MS, Univ. Illinois. 1992-1997: ETH Zurich.
I teach (taught) COM-102 "Advanced Information, Computation, Communication II" (2021-), EE-205 "Signals and Systems" (2015-19), COM-404 "Information Theory and Coding" (2016), COM-406 "Foundations of Data Science" (2017-19, 2021, 2023-), COM-421 "Statistical Neuroscience" (2012, 2014), COM-510 "Advanced Digital Communication" (2011-15), COM-621 "Advanced Topics in Information Theory" (2021,2023)
Previously at the University of California, Berkeley, I taught EE-120 "Signals and Systems" (2003-05), EE-121 "Communications" (2005), EE-123 "Signal Processing" (2007), EE-224A "Communications" (2010), EE-225A "Signal Processing" (2006-08), EE-226A "Random Processes" (2009), EE-290S "Advanced Information Theory (2004, 2006), MCB-262 "Theoretical Neuroscience" (2003, 05, 07, 09).
I have served as Technical Program Co-Chair for the IEEE International Symposium on Information Theory in 2010 (Austin, TX, USA) and in 2021 (Melbourne, Australia).