Xueyan Niu

Followers

Following

Public Views

Xueyan Niu received the B.S. degree in mathematics and applied mathematics from Peking University, Beijing, China, in 2016, and the Ph.D. degree in operations research from Purdue University, West Lafayette, IN, USA, in 2021. She was a Researcher with the Cognitive Computing Laboratory, Baidu Research, in 2021. Since 2022, she has been a Principal Engineer with the Theory Laboratory, Central Research Institute, 2012 Laboratories, Huawei Technologies Company Ltd. Her current research interests include information theory, machine learning, and semantic communications. Dr. Niu was a recipient of the EECS Rising Star Award from UC Berkeley in 2020. Her research in partial information decomposition has been recognized with the Outstanding Early Career Researcher Paper Award at 2020 IEEE International Symposium on Information Theory and Its Applications. She is the Guest Editor of the Special Issue on Goal-Oriented Semantic Communication and Networking for IEEE Network magazine.

less

Interests

Uploads

Papers by Xueyan Niu

Conditional Rate-Distortion-Perception Trade-Off

arXiv (Cornell University), May 16, 2023

Recent advances in machine learning-aided lossy compression are incorporating perceptual fidelity... more Recent advances in machine learning-aided lossy compression are incorporating perceptual fidelity into the ratedistortion theory. In this paper, we study the rate-distortionperception trade-off when the perceptual quality is measured by the total variation distance between the empirical and product distributions of the discrete memoryless source and its reconstruction. We consider the general setting, where two types of resources are available at both the encoder and decoder: a common side information sequence, correlated with the source sequence, and common randomness. We consider both the strong perceptual constraint and the weaker empirical perceptual constraint. The required communication rate for achieving the distortion and empirical perceptual constraint is the minimum conditional mutual information, and similar result holds for strong perceptual constraint when sufficient common randomness is provided and the output along with the side information is constraint to an independent and identically distributed sequence.

Download

A Hybrid Wireless Image Transmission Scheme with Diffusion

arXiv (Cornell University), Aug 15, 2023

We propose a hybrid joint source-channel coding (JSCC) scheme, in which the conventional digital ... more We propose a hybrid joint source-channel coding (JSCC) scheme, in which the conventional digital communication scheme is complemented with a generative refinement component to improve the perceptual quality of the reconstruction. The input image is decomposed into two components: the first is a coarse compressed version, and is transmitted following the conventional separation based approach. An additional component is obtained through the diffusion process by adding independent Gaussian noise to the input image, and is transmitted using DeepJSCC. The decoder combines the two signals to produce a high quality reconstruction of the source. Experimental results show that the hybrid design provides bandwidth savings and enables graceful performance improvement as the channel quality improves.

Download

Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory

arXiv (Cornell University), May 14, 2024

Increasing the size of a Transformer does not always lead to enhanced performance. This phenomeno... more Increasing the size of a Transformer does not always lead to enhanced performance. This phenomenon cannot be explained by the empirical scaling laws. Furthermore, the model's enhanced performance is closely associated with its memorization of the training samples. We present a theoretical framework that sheds light on the memorization during pretraining of transformer-based language models. We model the behavior of Transformers with associative memories using Hopfield networks, such that each transformer block effectively conducts an approximate nearest-neighbor search. In particular, the energy function in modern continuous Hopfield networks serves as an explanation for the attention mechanism, which we approximate with a distance-based energy function. By observing that the softmax function corresponds to the gradient of the LogSumExp function in the energy, and employing the majorization-minimization technique, we construct a global energy function designed to capture the layered architecture. We demonstrate a dependency between the model size and the dataset size for the model to achieve optimal performance, and we show that the achievable cross-entropy loss is bounded from below.

Download

Computation and Critical Transitions of Rate-Distortion-Perception Functions With Wasserstein Barycenter

arXiv (Cornell University), Apr 6, 2024

Blahut–Arimoto Algorithms for Inner and Outer Bounds on Capacity Regions of Broadcast Channels

Entropy, Feb 20, 2024

Extreme Video Compression with Pre-trained Diffusion Models

arXiv (Cornell University), Feb 13, 2024

Diffusion models have achieved remarkable success in generating high quality image and video data... more Diffusion models have achieved remarkable success in generating high quality image and video data. More recently, they have also been used for image compression with high perceptual quality. In this paper, we present a novel approach to extreme video compression leveraging the predictive power of diffusion-based generative models at the decoder. The conditional diffusion model takes several neural compressed frames and generates subsequent frames. When the reconstruction quality drops below the desired level, new frames are encoded to restart prediction. The entire video is sequentially encoded to achieve a visually pleasing reconstruction, considering perceptual quality metrics such as the learned perceptual image patch similarity (LPIPS) and the Fréchet video distance (FVD), at bit rates as low as 0.02 bits per pixel (bpp). Experimental results demonstrate the effectiveness of the proposed scheme compared to standard codecs such as H.264 and H.265 in the low bpp regime. The results showcase the potential of exploiting the temporal relations in video data using generative models. Code is available at: Compression-With-Prediction-Using-Pre-trainded-Diffusion-Models-

Download

High Perceptual Quality Wireless Image Delivery with Denoising Diffusion Models

arXiv (Cornell University), Sep 26, 2023

We consider the image transmission problem over a noisy wireless channel via deep learning-based ... more We consider the image transmission problem over a noisy wireless channel via deep learning-based joint source-channel coding (DeepJSCC) along with a denoising diffusion probabilistic model (DDPM) at the receiver. Specifically, we are interested in the perception-distortion trade-off in the practical finite block length regime, in which separate source and channel coding can be highly suboptimal. We introduce a novel scheme, where the conventional DeepJSCC encoder targets transmitting a lower resolution version of the image, which later can be refined thanks to the generative model available at the receiver. In particular, we utilize the range-null space decomposition of the target image; DeepJSCC transmits the range-space of the image, while DDPM progressively refines its null space contents. Through extensive experiments, we demonstrate significant improvements in distortion and perceptual quality of reconstructed images compared to standard DeepJSCC and the state-of-the-art generative learning-based method.

Download

Characterization of the Rate Region for Multi-Source Multicast Network Coding

Extending Context Window of Large Language Models via Semantic Compression

arXiv (Cornell University), Dec 14, 2023

Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text... more Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint restricts their applicability in scenarios involving long texts. We propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer, without incurring significant computational costs or requiring fine-tuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

Download

Generalizing Körner’s graph entropy to graphons

European Journal of Combinatorics

A Hybrid Wireless Image Transmission Scheme with Diffusion

2023 IEEE 24th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)

A Measure of Synergy, Redundancy, and Unique Information using Information Geometry

It is well known that joint interactions between agents can be described qualitatively as having ... more It is well known that joint interactions between agents can be described qualitatively as having synergistic, unique, and redundant components. In recent years, there have been renewed efforts to decompose mutual information, a general, non-parametric measure of joint interactions, into constituent parts. We propose a novel, non-negative decomposition of mutual information between two sources and a target variable. The decomposition is for the exponential family, and thus can be applied to a broad range of distributions. We also show that values from our decomposition arise naturally from testing hypotheses of conditional dependence. We demonstrate the method numerically using standard binary logic gates and Gaussian channels, as well as apply the method to investigate redundancy between brain regions using an fMRI-based image classification data-set.

Learning Cluster Causal Diagrams: An Information-Theoretic Approach

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, Jul 1, 2022

Unobserved confounding is the main obstacle to causal effect estimation from observational data. ... more Unobserved confounding is the main obstacle to causal effect estimation from observational data. Instrumental variables (IVs) are widely used for causal effect estimation when there exist latent confounders. With the standard IV method, when a given IV is valid, unbiased estimation can be obtained, but the validity requirement on a standard IV is strict and untestable. Conditional IVs have been proposed to relax the requirement of standard IVs by conditioning on a set of observed variables (known as a conditioning set for a conditional IV). However, the criterion for finding a conditioning set for a conditional IV needs a directed acyclic graph (DAG) representing the causal relationships of both observed and unobserved variables. This makes it challenging to discover a conditioning set directly from data. In this paper, by leveraging maximal ancestral graphs (MAGs) for causal inference with latent variables, we study the graphical properties of ancestral IVs, a type of conditional IVs using MAGs, and develop the theory to support data-driven discovery of the conditioning set for a given ancestral IV in data under the pretreatment variable assumption. Based on the theory, we develop an algorithm for unbiased causal effect estimation with a given ancestral IV and observational data. Extensive experiments on synthetic and realworld datasets demonstrate the performance of the algorithm in comparison with existing IV methods.

Download

Information Flow in Markov Chains

2021 60th IEEE Conference on Decision and Control (CDC), Dec 14, 2021

We consider the problem of characterizing the flow of information in stochastic systems. Recently... more We consider the problem of characterizing the flow of information in stochastic systems. Recently, several measures of partial information decomposition (PID) have been proposed which, for a fixed target variable, can distinguish unique, redundant, and synergistic contributions from the predictor variables. We study how each of those partial informations travel in a Markov chain, entering at one variable, passing through several variables, and eventually exiting downstream. Our work is agnostic to specific partial information decomposition (PID) measures. We investigate partial information flow among variables relating to overflow events in a river system.

Computation of Rate-Distortion-Perception Functions With Wasserstein Barycenter

arXiv (Cornell University), Apr 27, 2023

The nascent field of Rate-Distortion-Perception (RDP) theory is seeing a surge of research intere... more The nascent field of Rate-Distortion-Perception (RDP) theory is seeing a surge of research interest due to the application of machine learning techniques in the area of lossy compression. The information RDP function characterizes the three-way trade-off between description rate, average distortion, and perceptual quality measured by discrepancy between probability distributions. However, computing RDP functions has been a challenge due to the introduction of the perceptual constraint, and existing research often resorts to data-driven methods. In this paper, we show that the information RDP function can be transformed into a Wasserstein Barycenter problem. The nonstrictly convexity brought by the perceptual constraint can be regularized by an entropy regularization term. We prove that the entropy regularized model converges to the original problem. Furthermore, we propose an alternating iteration method based on the Sinkhorn algorithm to numerically solve the regularized optimization problem. Experimental results demonstrate the efficiency and accuracy of the proposed algorithm.

Download

Conditional graph entropy as an alternating minimization problem

IEEE Transactions on Information Theory

Conditional Rate-Distortion-Perception Trade-Off

2023 IEEE International Symposium on Information Theory (ISIT)

$Research paper thumbnail of Generalizing K\"orner's graph entropy to graphons$

Generalizing K\"orner's graph entropy to graphons

arXiv (Cornell University), Aug 31, 2022

Körner introduced the notion of graph entropy in 1973 as the minimal code rate of a natural codin... more Körner introduced the notion of graph entropy in 1973 as the minimal code rate of a natural coding problem where not all pairs of letters can be distinguished in the alphabet. Later it turned out that it can be expressed as the solution of a minimization problem over the so-called vertex-packing polytope. In this paper we generalize this notion to graphons. We show that the analogous minimization problem provides an upper bound for graphon entropy. We also give a lower bound in the shape of a maximization problem. The main result of the paper is that for most graphons these two bounds actually coincide and hence precisely determine the entropy in question. Furthermore, graphon entropy has a nice connection to the fractional chromatic number and the fractional clique number.

Download

Conditional graph entropy as an alternating minimization problem

arXiv (Cornell University), Sep 1, 2022

Conditional graph entropy is known to be the minimal rate for a natural functional compression pr... more Conditional graph entropy is known to be the minimal rate for a natural functional compression problem with side information at the receiver. In this paper we show that it can be formulated as an alternating minimization problem, which gives rise to a simple iterative algorithm for numerically computing (conditional) graph entropy. This also leads to a new formula which shows that conditional graph entropy is part of a more general framework: the solution of an optimization problem over a convex corner. In the special case of graph entropy (i.e., unconditioned version) this was known due to Csiszár, Körner, Lovász, Marton, and Simonyi. In that case the role of the convex corner was played by the so-called vertex packing polytope. In the conditional version it is a more intricate convex body but the function to minimize is the same. Furthermore, we describe a dual problem that leads to an optimality check and an error bound for the iterative algorithm.

Download

Learning Cluster Causal Diagrams: An Information-Theoretic Approach

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence

Many real-world phenomena arise from causal relationships among a set of variables. As a powerful... more Many real-world phenomena arise from causal relationships among a set of variables. As a powerful tool, Bayesian Network (BN) has been successful in describing high-dimensional distributions. However, the faithfulness condition, enforced in most BN learning algorithms, is violated in the settings where multiple variables synergistically affect the outcome (i.e., with polyadic dependencies). Building upon recent development in cluster causal diagrams (C-DAGs), we initiate the formal study of learning C-DAGs from observational data to relax the faithfulness condition. We propose a new scoring function, the Clustering Information Criterion (CIC), based on information-theoretic measures that represent various complex interactions among variables. The CIC score also contains a penalization of the model complexity under the minimum description length principle. We further provide a searching strategy to learn structures of high scores. Experiments on both synthetic and real data support t...

Download

Xueyan Niu

Uploads

Papers by Xueyan Niu

Log In