T. Basar

University of Illinois at Urbana-Champaign, Electrical and Computer Engineering, Faculty Member

Followers

Following

Co-authors

Public Views

Interests

Uploads

Papers by T. Basar

Finite-Sample Analysis for Decentralized Batch Multiagent Reinforcement Learning With Networked Agents

IEEE Transactions on Automatic Control, Dec 1, 2021

Despite the increasing interest in multi-agent reinforcement learning (MARL) in multiple communit... more Despite the increasing interest in multi-agent reinforcement learning (MARL) in multiple communities, understanding its theoretical foundation has long been recognized as a challenging problem. In this work, we address this problem by providing a finite-sample analysis for decentralized batch MARL with networked agents. Specifically, we consider two decentralized MARL settings, where teams of agents are connected by time-varying communication networks, and either collaborate or compete in a zero-sum game setting, without any central controller. These settings cover many conventional MARL settings in the literature. For both settings, we develop batch MARL algorithms that can be implemented in a decentralized fashion, and quantify the finite-sample errors of the estimated action-value functions. Our error analysis captures how the function class, the number of samples within each iteration, and the number of iterations determine the statistical accuracy of the proposed algorithms. Our results, compared to the finite-sample bounds for single-agent RL, involve additional error terms caused by decentralized computation, which is inherent in our decentralized MARL setting. This work appears to be the first finite-sample analysis for batch MARL, a step towards rigorous theoretical understanding of general MARL algorithms in the finite-sample regime.

Download

Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions

Mathematics of Operations Research, Aug 1, 2019

Establishing the existence of Nash equilibria for partially observed stochastic dynamic games is ... more Establishing the existence of Nash equilibria for partially observed stochastic dynamic games is known to be quite challenging, with the difficulties stemming from the noisy nature of the measurements available to individual players (agents) and the decentralized nature of this information. When the number of players is sufficiently large and the interactions among agents is of the mean-field type, one way to overcome this challenge is to investigate the infinite-population limit of the problem, which leads to a mean-field game. In this paper, we consider discrete-time partially observed mean-field games with infinite-horizon discounted cost criteria. Using the technique of converting the original partially observed stochastic control problem to a fully observed one on the belief space and the dynamic programming principle, we establish the existence of Nash equilibria for these game models under very mild technical conditions. Then, we show that the mean-field equilibrium policy, when adopted by each agent, forms an approximate Nash equilibrium for games with sufficiently many agents.

Download

Policy Optimization Provably Converges to Nash Equilibria in Zero-Sum Linear Quadratic Games

arXiv (Cornell University), May 31, 2019

We study the global convergence of policy optimization for finding the Nash equilibria (NE) in ze... more We study the global convergence of policy optimization for finding the Nash equilibria (NE) in zero-sum linear quadratic (LQ) games. To this end, we first investigate the landscape of LQ games, viewing it as a nonconvex-nonconcave saddle-point problem in the policy space. Specifically, we show that despite its nonconvexity and nonconcavity, zero-sum LQ games have the property that the stationary point of the objective function with respect to the linear feedback control policies constitutes the NE of the game. Building upon this, we develop three projected nested-gradient methods that are guaranteed to converge to the NE of the game. Moreover, we show that all of these algorithms enjoy both globally sublinear and locally linear convergence rates. Simulation results are also provided to illustrate the satisfactory convergence properties of the algorithms. To the best of our knowledge, this work appears to be the first one to investigate the optimization landscape of LQ games, and provably show the convergence of policy optimization methods to the Nash equilibria. Our work serves as an initial step toward understanding the theoretical aspects of policy-based reinforcement learning algorithms for zero-sum Markov games in general.

Download

Graph-theoretic approach for connectivity maintenance in mobile networks in the presence of a jammer

49th IEEE Conference on Decision and Control (CDC), 2010

Abstract In this work, we analyze the problem of maintain-ing connectivity in a network of mobil... more

Memories and Lessons from Adaptive Control

H∞-Optimal Boundary Control of Hyperbolic Systems with Sampled Measurements1

Weighted Age of Information based Scheduling for Large Population Games on Networks

arXiv (Cornell University), Sep 26, 2022

In this paper, we consider a discrete-time multi-agent system involving N cost-coupled networked ... more In this paper, we consider a discrete-time multi-agent system involving N cost-coupled networked rational agents solving a consensus problem and a central Base Station (BS), scheduling agent communications over a network. Due to a hard bandwidth constraint on the number of transmissions through the network, at most R d < N agents can concurrently access their state information through the network. Under standard assumptions on the information structure of the agents and the BS, we first show that the control actions of the agents are free of any dual effect, allowing for separation between estimation and control problems at each agent. Next, we propose a weighted age of information (WAoI) metric for the scheduling problem of the BS, where the weights depend on the estimation error of the agents. The BS aims to find the optimum scheduling policy that minimizes the WAoI, subject to the hard bandwidth constraint. Since this problem is NPhard, we first relax the hard constraint to a soft updaterate constraint, and then compute an optimal policy for the relaxed problem by reformulating it into a Markov Decision Process (MDP). This then inspires a sub-optimal policy for the bandwidth constrained problem, which is shown to approach the optimal policy as N → ∞. Next, we solve the consensus problem using the mean-field game framework wherein we first design decentralized control policies for a limiting case of the N-agent system (as N → ∞). By explicitly constructing the mean-field system, we prove the existence and uniqueness of the mean-field equilibrium. Consequently, we show that the obtained equilibrium policies constitute an ǫ-Nash equilibrium for the finite agent system. Finally, we validate the performance of both the scheduling and the control policies through numerical simulations.

Download

Optimal strategies to evade jamming in heterogeneous mobile networks

In this paper, we consider a scenario in which a mobile intruder jams the communication network i... more In this paper, we consider a scenario in which a mobile intruder jams the communication network in a vehicular formation. We formulate the problem as a zero-sum pursuitevasion game with players possessing heterogeneous dynamics. We use Isaacs' approach to derive the necessary conditions to arrive at the equations governing the saddle-point strategies of the team of players. Sourabh Bhattacharya and Tamer Başar are with the Department of Electrical and Computer Engineering,

Download

A Stochastic Game Model for Jamming in Multi-Channel Cognitive Radio Systems

2010 IEEE International Conference on Communications, 2010

The security issue in collaborative sensing in cognitive radio networks can be modeled as attacke... more The security issue in collaborative sensing in cognitive radio networks can be modeled as attackers and secondary users in a jamming and anti-jamming scenario. In this paper, we introduce a stochastic zero-sum game model to study the strategies. Primary users, secondary users and jammers are the three types of agents in the system. The primary users dictate the system states

Paradigms for Robustness in Controller and Filter Designs

Modeling and Control of Economic Systems 2001, 2003

We provide an overview of robustness issues that arise in controller and filter designs for dynam... more We provide an overview of robustness issues that arise in controller and filter designs for dynamic systems subject to various types of uncertainty. Both linear and nonlinear systems are treated, with the nominal system being deterministic, stochastic or piecewise deterministic. The performance indices adopted cover a broad spectrum of scenarios, including risk-neutral, risk-sensitive, and worst-case formulations.

Download

Switching Strategies for Linear Feedback Stabilization with Sparsified State Measurements

In this paper, we address the problem of stabilization in continuous time linear dynamical system... more In this paper, we address the problem of stabilization in continuous time linear dynamical systems using state feedback when compressive sampling techniques are used for state measurement and reconstruction. In [5], we had introduced the concept of using l1 reconstruction technique, commonly used in sparse data reconstruction, for state measurement and estimation in a discrete time linear system. In this work, we extend the previous scenario to analyse continuous time linear systems. We investigate the effect of switching within a set of sparsifiers, introduced in [5], on the stability of a linear plant in continuous time settings. Initially, we analyze the problem of stabilization in low dimensional systems, following which we generalize the results to address the problem of stabilization in systems of arbitrary dimensions.

Download

Coordinated Sci. Lab., Univ. of Illinois at Urbana-Champaign, Urbana, IL, USA

Network Formation Games Among Relay Stations in Next Generation Wireless Networks

IEEE Transactions on Communications, 2011

The introduction of relay station (RS) nodes is a key feature in next generation wireless network... more The introduction of relay station (RS) nodes is a key feature in next generation wireless networks such as 3GPP's long term evolution advanced (LTE-Advanced), or the forthcoming IEEE 802.16j WiMAX standard. This paper presents, using game theory, a novel approach for the formation of the tree architecture that connects the RSs and their serving base station in the uplink of the next generation wireless multi-hop systems. Unlike existing literature which mainly focused on performance analysis, we propose a distributed algorithm for studying the structure and dynamics of the network. We formulate a network formation game among the RSs whereby each RS aims to maximize a crosslayer utility function that takes into account the benefit from cooperative transmission, in terms of reduced bit error rate, and the costs in terms of the delay due to multi-hop transmission. For forming the tree structure, a distributed myopic algorithm is devised. Using the proposed algorithm, each RS can individually select the path that connects it to the BS through other RSs while optimizing its utility. We show the convergence of the algorithm into a Nash tree network, and we study how the RSs can adapt the network's topology to environmental changes such as mobility or the deployment of new mobile stations. Simulation results show that the proposed algorithm presents significant gains in terms of average utility per mobile station which is at least 17.1% better relatively to the case with no RSs and reaches up to 40.3% improvement compared to a nearest neighbor algorithm (for a network with 10 RSs). The results also show that the average number of hops does not exceed 3 even for a network with up to 25 RSs.

Download

Control Over Noisy Forward and Reverse Channels

IEEE Transactions on Automatic Control, 2011

We consider the problem of remotely controlling a continuous-time linear time-invariant system dr... more We consider the problem of remotely controlling a continuous-time linear time-invariant system driven by Brownian motion process, when communication takes place over noisy memoryless discrete-or continuous-alphabet channels. What makes this class of remote control problems different from most of the previously studied models is the presence of noise in both the forward channel (connecting sensors to the controller) and the reverse channel (connecting the controller to the plant). For stability of the closed-loop system, we look for the existence of an invariant distribution for the state, for which we show that it is necessary that the entire control space and the state space be encoded, and that the reverse channel be at least as reliable as the forward channel. We obtain necessary conditions and sufficient conditions on the channels and the controllers for stabilizability. Using properties of the underlying sampled Markov chain, we show that under variable-length coding and some realistic channel conditions, stability can be achieved over discrete-alphabet channels even if the entire state and control spaces are to be encoded and the number of bits that can be transmitted per unit time is strictly bounded. For control over continuous-alphabet channels, however, a variable rate scheme is not necessary. We also show that memoryless policies are rate-efficient for Gaussian channels.

Download

Communication Constraints for Decentralized Stabilizability With Time-Invariant Policies

IEEE Transactions on Automatic Control, 2007

In this note, we study communication requirements for decentralized control schemes when sensors ... more In this note, we study communication requirements for decentralized control schemes when sensors and controllers are connected via noiseless digital channels and system noises are bounded. Under various information structures, and within the class of time-invariant encoding and control policies, we quantify achievable rates, and obtain fixed-rate and delayless encoders and decoders that achieve boundedness in the state estimation errors at the controllers. We use distributed coding theoretic approaches to show that when the sensors do not collaborate, it is sufficient for the controllers to share information amongst themselves but not with the sensors. Sensor collaboration, however, leads to improvement in rate performance.

Download

Randomized algorithms for synthesis of switching rules for multimodal systems

IEEE Transactions on Automatic Control, 2005

Download

Coalitional Games in Partition Form for Joint Spectrum Sensing and Access in Cognitive Radio Networks

Unlicensed secondary users (SUs) in cognitive radio networks are subject to an inherent tradeoff ... more Unlicensed secondary users (SUs) in cognitive radio networks are subject to an inherent tradeoff between spectrum sensing and spectrum access. Although each SU has an incentive to sense the primary user (PU) channels for locating spectrum holes, this exploration of the spectrum can come at the expense of a shorter transmission time, and, hence, a possibly smaller capacity for data transmission. This paper investigates the impact of this tradeoff on the cooperative strategies of a network of SUs that seek to cooperate in order to improve their view of the spectrum (sensing), reduce the possibility of interference among each other, and improve their transmission capacity (access). The problem is modeled as a coalitional game in partition form and an algorithm for coalition formation is proposed. Using the proposed algorithm, the SUs can make individual distributed decisions to join or leave a coalition while maximizing their utilities which capture the average time spent for sensing a...

Download

Stochastic stability of singularly perturbed nonlinear systems

Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228)

The stability of a nonlinear stochastic dynamic system with singular perturbations is considered.... more The stability of a nonlinear stochastic dynamic system with singular perturbations is considered. Based on the notion of stochastic input-to-state stability and using time scale decomposition, a result of the "total stability" type is obtained, i.e. if the fast subsystem and the slow subsystem are both input-to-state stable with respect to disturbances, then this property continues to hold for the full-order system as long as the singular perturbation parameter is sufficiently small and a stochastic small gain condition is satisfied. The result is general in that it holds for a broad class of disturbances, and resembles similar results for deterministic systems.

CDMA uplink power control as a noncooperative game

Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No.01CH37228)

We present a game-theoretic treatment of distributed power control in CDMA wireless systems. We m... more We present a game-theoretic treatment of distributed power control in CDMA wireless systems. We make use of the conceptual framework of noncooperative game theory to obtain a distributed and market-based control mechanism. Thus, we address not only the power control problem, but also pricing and allocation of a single resource among several users. A cost function is introduced as the difference between the pricing and utility functions, and the existence of a unique Nash equilibrium is established. In addition, two update algorithms, namely, parallel update and random update, are shown to be globally stable under specific conditions. Convergence properties and robustness of each algorithm are also studied through extensive simulations.

Download

Structural properties of minimax policies for a class of differential games arising in nonlinear H/sup ∞/-control and filtering

Proceedings of 32nd IEEE Conference on Decision and Control

This paper introduces, in precise mathematical terms, two properties (named, certainty equivalenc... more This paper introduces, in precise mathematical terms, two properties (named, certainty equivalence and generalized certainty equivalence) that nonlinear minimax controller problems might possess. The certainty equivalence is a generalization of the one introduced earlier in [2] and [3], which applies to problems where the "worst-case disturbance" may not be unique (but the worst-case state trajectory is). The generalized certainty equivalence, on the other hand, extends this to accommodate nonunique worst-case state trajectories, and leads to the construction of controllers that guarantee a bounded upper value for the underlying game. The paper also shows that for a large class of games (and under certain conditions) certaintyequivalent (as well as generalized certainty-equivalent) controllers admit (infinite-dimensional) estimator (Kalman-filter) structures, where the estimator gain depends on the state of the estimator. These results are then applied to the nonlinear minimax filtering problem, which is treated here as a special case of the general control problem.

Download

T. Basar

Uploads

Papers by T. Basar

Log In