Figure 4 Q-learning algorithm representation for the online stage.
Related Figures (10)
FIGURE 1. Environmental setup and system model. We formulate the UAV localization problem as an optimiza- tion problem to minimize the sum of the least square estimate for the errors of the measured distances between the UAV and the m = 4 gNBs. This problem can be written as iil. PROBLEM FORMULATION TABLE 1. Computational complexity illustration for algorithm steps. V. THE PROPOSED DEEP LEARNING BASED APPROACH FIGURE 2. n-layer feed-forward neural network architecture. Algorithm 2 Deep Supervised Learning Approach FIGURE 3. Critic network architecture. In this subsection, we detail the overall algorithm and compu- tational complexity for the proposed reinforcement learning based approach. As shown in Fig. 4, the current state 16 x 1 vector, Ss, is input to the agent. The agent multiplies the state vector, s, by a 1 xn unit vector J. The resulting 16 x n vector is then applied to the addition layer of critic network to evaluate the approximate Q-value for each state-action pair. The action with the maximum Q-value is then selected as the most likely estimate for the UAV location. The proposed reinforcement learning based approach is summarized in Algorithm 3. TABLE 2. Environmental and system parameters. A. THE CARRIER PHASE APPROACH FIGURE 5. Optimization-based localization results (a) Mean localization error for exhaustive search vs. Nelder-Mead; (b) Maximum localization error as compared to theoretical bounds for exhaustive search vs. Nelder-Mead. and v;(k) is the zero mean gaussian measurement noise. The carrier phase observable can be re-parametrized in terms of UAV and BS positions as follows The above estimation ignores the initial measurement noise. However, the authors propose to average several mea- surements for the total duration in which the GPS is initially available. To estimate the common bias term cét (k), the authors propose to lump the N BS; clocks into clusters, each of size Nj where N = iy N_ and L is the total number of clusters. Extending this approach to our application scenario to localize the UAV through measurements from four base stations, we let N = 4 and L < 2. Note that, given the 2-D position of the UAV is being estimated with L clock clus- ters, the number of clusters, L, cannot exceed N-2. Next, the authors utilize a weighted non-linear least squares (WNLS) estimator to estimate the UAV location given by FIGURE 6. Deep learning-based localization results (a) Mean localization error vs. no. of nodes for a 1-Layer NN; (b) Mean localization error vs. no. of Epochs for a 2-Layer NN; (c) Mean localization error vs. no. of layers; (d) Mean localization error vs. data size. FIGURE 7. Reinforcement learning-based localization results (a) Mean localization error vs. number of layers; (b) Mean localization error vs. experience replay batch size; (c) Mean localization error vs. number of nodes per hidden layer; (d) Mean localization error vs. action space discretization step size. FIGURE 8. Overall analysis (a) Mean localization error vs. gNB-UAV placement scenario; (b) Time complexity vs. problem dimension.
