tug-of-war dynamics

The rapid expansion of wireless networks demands efficient spectrum allocation. Dynamic Spectrum Sharing (DSS) is a technology that allows multiple wireless networks to share the same frequency spectrum dynamically. It is an effective technique for optimizing spectrum use, particularly in heterogeneous environments where multiple wireless technologies with diverse spectrum access requirements coexist, often leading to interference challenges and increased spectrum competition. This research proposes an enhanced DSS technique based on Deep Reinforcement Learning (DRL). The proposed method enables an effective sharing of the available spectrum between two access technologies, namely Long Term Evolution (LTE) and Narrowband IoT (NB-IoT). The study optimizes throughput through DRL methods, including Deep Q-Networks (DQN), conducting experiments in three phases: LTE-DRL coexistence, NB-IoT-DRL coexistence, and LTE-NB-IoT coexistence. Results show that deep learning enhances the LTE-DRL system's convergence rate and throughput, achieving over 85% throughput with convergence times as low as 24 milliseconds (ms). The study highlights the trade-offs between parameters such as probabilities (arrival, successful transmission, and retransmission), packet expiry duration, learning rate, discount factor, fairness index, and the neural network architecture as well as the parameters' impact on the overall system throughput. NB-IoT coexistence with DRL shows similar results with a slight decrement in throughput and negligibly longer convergence rate, while the coexistence of LTE and NB-IoT results in throughput of around 70% for each of the LTE and NB-IoT systems due to increased spectrum competition and increased complexity of the operating environment. This work offers insights into optimizing spectrum sharing using DRL and underscores the balance between various parameters for efficient spectrum management.

UnderWater (UW) Acoustic networks face unique challenges due to limited bandwidth, high latency, and dynamic channel conditions, necessitating adaptive communication protocols to optimize performance under strict energy constraints. Modulation schemes play a crucial role in determining the efficiency and reliability of these networks; dynamically adjusting modulation depending on channel conditions can significantly enhance network performance. While Machine Learning algorithms offer valuable solutions for real-time adaptation, many existing methods are based on deep learning, which often demands computational resources beyond the capabilities of typical UW devices. In contrast, Multi-Armed Bandit (MAB) algorithms offer a simpler yet effective solution, well-suited for environments with limited computational resources. In this paper, we present AMUSE, a scalable and efficient framework designed to leverage the MAB approach for dynamic modulation selection, while enabling the optimization of various key performance metrics. Specifically, to illustrate the high level of flexibility of AMUSE in addressing multi-objective optimization, we here focus on the trade-off of Packet Error Rate (PER) and energy consumption across changing conditions, so as to make both reliability and energy efficiency the basis of the modulation adaptation decision-making process. Through extensive simulation in the DESERT simulator, we evaluate AMUSE performance against other state-of-the-art algorithms based on Deep Reinforcement Learning (DRL). Despite its simple design, AMUSE proves to be more efficient and responsive than the baselines, making it a powerful solution for improving UW communication performance. The results show that, in spite of the lightweight nature of AMUSE, our framework is able to outperform the DRL baselines by achieving an improvement of up to 23.64% in the network PER, and up to 80.65% in energy saving.

Providing acceptable Quality of Experience (QoE) in Wireless Local Area Network (WLAN) is very difficult: home networks are managed by non-technical people, and the proprietary management solutions of enterprise networks usually do not incorporate QoE mechanisms. Due to these difficulties, automatic QoE management mechanisms are welcome. This paper presents control loops capable of changing the power and transmission channels in the WLAN, based on Software Defined Wireless Networks and reinforcement learning, in order to improve user satisfaction for Web applications. A prototype evaluates our proposal in three case studies with a web browsing application, in which several access points are controlled by a central controller or by independent controllers. Our results show that the control loop can improve the Mean Opinion Score (MOS) by at least 4 % in the worst case, and 167 % in the best case, thus benefiting the user. Further, the control loop also reduced in page load time by 25...

In recent years, the demand for new applications using various Internet of Things (IoT) devices has led to an increase in the number of devices connected to wireless networks. However, owing to the limitation of available frequency resources for IoT devices, the degradation of the communication quality caused by channel congestion is a practical problem in developing IoT technology. Many IoT devices have hardware and software limitations that prevent centralized channel allocation, and congestion is even more severe in massive IoT networks without a central controller. Therefore, developing a distributed and sophisticated channel selection algorithm is necessary. In previous studies, the channel selection of each IoT device was modeled as a multi-armed bandit (MAB) problem, and a wireless channel selection method based on the MAB algorithm, which is a simple reinforcement learning, was proposed. In particular, it has been shown that the MAB algorithm of tug-of-war (TOW) dynamics can efficiently select channels with much lower computational complexity and power compared with other reinforcement learning-based channelselection methods. This paper proposes a distributed channel selection method based on TOW dynamics in fully decentralized networks. We evaluate the effectiveness of the proposed method and other distributed channel-selection methods on the communication success rate in massive IoT networks by experiments and simulations. The results show that the proposed method improves the communication success rate more than other distributed channel selection methods even in a dense and dynamic network environment.

Dynamic channel selection is among the most important wireless communication elements in dynamically changing electromagnetic environments wherein, a user can experience improved communication quality by choosing a better channel. Multi-armed bandit (MAB) algorithms are a promising approach that resolve the trade-off between channel exploration and exploitation of enhanced communication quality. Ultrafast solution of MAB problems has been demonstrated by utilizing chaotically oscillating time series generated by semiconductor lasers. in this study, we experimentally demonstrate a MAB algorithm incorporating laser chaos time series in a wireless local area network (WLAN). Autonomous and adaptive dynamic channel selection is successfully demonstrated in an IEEE802.11a-based, four-channel WLAN. Although the laser chaos time series is arranged prior to the WLAN experiments, the results confirm the usefulness of ultrafast chaotic sequences for real wireless applications. In addition, we numerically examine the underlying adaptation mechanism of the significantly simplified MAB algorithm implemented in the present study compared with the previously reported chaos-based decision makers. This study provides a first step toward the application of ultrafast chaotic lasers for future high-performance wireless communication networks.

Non-Orthogonal Multiple Access is one of the most important technologies in 5G and Beyond 5G wireless communications, which improve system performance by power domain multiplexing. In realizing Non-Orthogonal Multiple Access, the pairing of multiple users is necessary where efficient principles are highly demanded in dynamically changing electromagnetic environments. In the meantime, ultrafast methods of solving multi-armed bandit problems have been developed using chaotic laser time series. In this paper, we consider the user pairing problem in Non-Orthogonal Multiple Access as a multi-armed bandit problem and propose an ultra-fast user pairing algorithm based on the laser chaos decision maker. We numerically demonstrate that the proposed scheme accomplishes higher throughputs compared with traditional user pairing algorithms, especially in cases with lower user density.

Non-terrestrial networks, composed of ground, air, and satellite communications, are considered one of the key components for the Beyond 5G/6G, and optical satellite communication is a fundamental technology to enable high-capacity communications. It is affected by interruptions of optical communications due to clouds on the communication link. A satellite can mitigate the interruption by switching its destination ground station to the other communication available station, though it brings additional delays in establishing optical links. In this study, we propose a ground station selection method using reinforcement learning algorithms to realize a fast and stable satellite-terrestrial optical communication system. We introduce two multi-armed bandit algorithms, Q-learning and Deep Q-learning, for the proposed method. We evaluate them using actual data of the optical satellite communication availability. Our simulation results show that the proposed method with deep Q-learning has the best average throughput. The proposed scheme efficiently follows changes in the state of communication links, and it becomes even better than fixed to ideal best link.

We propose a simple channel-allocation method based on tug-of-war (TOW) dynamics, combined with the time scheduling based on nonlinear oscillator synchronization to efficiently use of the space (channel) and time resources in wireless communications. This study demonstrates that synchronization groups, where each node selects a different channel, are non-uniformly distributed in phase space such that every distance between groups is larger than the area of influence. New type of self-organized spatiotemporal patterns can be formed for resource allocation according to channel rewards.

High-bandwidth irregular oscillations caused by optical time-delayed feedback subjected to the laser cavity, known as laser chaos, have been investigated for various engineering applications. Recently, a fast decision-making algorithm for a multi-arm bandit problem by utilizing laser chaos time series has been demonstrated. Furthermore, the arms order recognition of the reward expectation for each arm has been successfully developed by incorporating the notion of the confidence interval regarding the reward estimate. However, in previous studies, the verification was limited to numerical experiments; real-world demonstrations were not conducted. This study experimentally demonstrated that the arm-order recognition algorithm is successfully operated in channel order recognition in wireless communications while revising the original strategy to take into account the wireless application requirements. Such accurate arm rank recognition involving non-best arms would be useful for various real-world applications such as channel bonding, among others.

Mobile wireless communication demand has increased rapidly in recent years, and advanced mobile devices such as smartphones have been widely deployed. Those devices are generally capable of connecting various wireless networks such as cellular networks and wireless LANs. Due to the software and hardware constraint of mobile devices, it is necessary to develop efficient and lightweight algorithms to select wireless networks. Previous studies have shown that multi-armed bandit algorithms can efficiently select wireless channel in cognitive radio. In this paper, we propose an efficient wireless network selection technique by using a multi-armed bandit algorithm called tug-of-war (TOW). We implement the proposed algorithm to a wireless device and show the effectiveness of the proposed method by experimental demonstrations.

Recently, various mobile communication systems have been widely deployed, and mobile traffic is increasing. However, the bandwidth available for mobile communications is limited, hence the scarcity of radio resources in mobile communications is a serious problem. As an approach to solve this problem, cognitive wireless communication models have been proposed. These model search for vacant time slots in multi-channel wireless communication systems. Although previous studies have shown that frequency utilization efficiency can be improved by multi-armed bandit algorithms, channels are assumed to be independent. However, channels used in 2.4 GHz wireless LANs (such as IEEE802.11b or IEEE802.11g) are not independent because these channels overlap with adjacent channels. In this paper, we propose an extended multi-armed bandit algorithm that uses continuous-valued rewards, which is applicable to wireless communication systems with overlapping channels. We show the effectiveness of the proposed method by experimental demonstrations.

The wide spread of mobile communication devices has increased the opportunities to use wireless communication technologies, irrespective of one's geographical location. However communication quality deteriorates due to factors such as competition for scarce radio resources and interference among nearby devices. Cognitive radio technologies have been developed recently to conquer such difficulties. In this paper, we propose a wireless network optimization method using learning algorithms based on a control model known as cognitive cycle. We implement the proposed optimization method in wireless LANs and evaluate the throughput performance. The experimental results show the effectiveness of the proposed approach in a real environment.

Mobile wireless communication demand has increased rapidly in recent years, and advanced mobile devices such as smartphones have been widely deployed. Those devices are generally capable of connecting various wireless networks such as cellular networks and wireless LANs. Due to the software and hardware constraint of mobile devices, it is necessary to develop efficient and lightweight algorithms to select wireless networks. Previous studies have shown that multi-armed bandit algorithms can efficiently select wireless channel in cognitive radio. In this paper, we propose an efficient wireless network selection technique by using a multi-armed bandit algorithm called tug-of-war (TOW). We implement the proposed algorithm to a wireless device and show the effectiveness of the proposed method by experimental demonstrations.

Recently, various mobile communication systems have been widely deployed, and mobile traffic is increasing. However, the bandwidth available for mobile communications is limited, hence the scarcity of radio resources in mobile communications is a serious problem. As an approach to solve this problem, cognitive wireless communication models have been proposed. These model search for vacant time slots in multi-channel wireless communication systems. Although previous studies have shown that frequency utilization efficiency can be improved by multi-armed bandit algorithms, channels are assumed to be independent. However, channels used in 2.4 GHz wireless LANs (such as IEEE802.11b or IEEE802.11g) are not independent because these channels overlap with adjacent channels. In this paper, we propose an extended multi-armed bandit algorithm that uses continuous-valued rewards, which is applicable to wireless communication systems with overlapping channels. We show the effectiveness of the proposed method by experimental demonstrations.

Non-orthogonal multiple access schemes with grant free access have been recently highlighted as a prominent solution to meet the stringent requirements of massive machine-type communications (mMTCs). In particular, the multi-user shared access (MUSA) scheme has shown great potential to grant free access to the available resources. For the sake of simplicity, MUSA is generally conducted with the successive interference cancellation (SIC) receiver, which offers a low decoding complexity. However, this family of receivers requires sufficiently diversified received user powers in order to ensure the best performance and avoid the error propagation phenomenon. The power allocation has been considered as a complicated issue especially for a decentralized decision with a minimum signaling overhead. In this paper, we propose a novel algorithm for an autonomous power decision with a minimal overhead based on a tight approximation of the bit error probability (BEP) while considering the error...

Multi-armed bandit (MAB) models are a viable approach to describe the problem of best wireless network selection by a multi-Radio Access Technology (multi-RAT) device, with the goal of maximizing the quality perceived by the final user. The classical MAB model does not allow, however, to properly describe the problem of wireless network selection by a multi-RAT device, in which a device typically performs a set of measurements in order to collect information on available networks, before a selection takes place. The MAB model foresees in fact only one possible action for the player, which is the selection of one among different arms at each time step; existing arm selection algorithms thus mainly differ in the rule according to which a specific arm is selected. This work proposes a new MAB model, named measure-use-MAB (muMAB), aiming at providing a higher flexibility, and thus a better accuracy in describing the network selection problem. The muMAB model extends the classical MAB model in a twofold manner; first, it foresees two different actions: to measure and to use; second, it allows actions to span over multiple time steps. Two new algorithms designed to take advantage of the higher flexibility provided by the muMAB model are also introduced. The first one, referred to as measure-use-UCB1 (muUCB1) is derived from the well known UCB1 algorithm, while the second one, referred to as Measure with Logarithmic Interval (MLI), is appositely designed for the new model so to take advantage of the new measure action, while aggressively using the best arm. The new algorithms are compared against existing ones from the literature in the context of the muMAB model, by means of computer simulations using both synthetic and captured data. Results show that the performance of the algorithms heavily depends on the Probability Density Function (PDF) of the reward received on each arm, with different algorithms leading to the best performance depending on the PDF. Results highlight, however, that as the ratio between the time required for using an arm and the time required to measure increases, the proposed algorithms guarantee the best performance, with muUCB1 emerging as the best candidate when the arms are characterized by similar mean rewards, and MLI prevailing when an arm is significantly more rewarding than others. This calls thus for the introduction of an adaptive approach capable of adjusting the behavior of the algorithm or of switching algorithm altogether, depending on the acquired knowledge on the PDF of the reward on each arm.

Long-Range (LoRa) devices have been deployed in many Internet of Things (IoT) applications due to their ability to communicate over long distances with low power consumption. The scalability and communication performance of the LoRa systems are highly dependent on the spreading factor (SF) and channel allocations. In particular, it is important to set the SF appropriately according to the distance between the LoRa device and the gateway since the signal reception sensitivity and bit rate depend on the used SF, which are in a trade-off relationship. In addition, considering the surge in the number of LoRa devices recently, the scalability of LoRa systems is also greatly affected by the channels that the LoRa devices use for communications. It was demonstrated that the lightweight decentralized learning-based joint channel and SF-selection methods can make appropriate decisions with low computational complexity and power consumption in our previous study. However, the effect of the lo...

Long-Range (LoRa) devices have been deployed in many Internet of Things (IoT) applications due to their ability to communicate over long distances with low power consumption. The scalability and communication performance of the LoRa systems are highly dependent on the spreading factor (SF) and channel allocations. In particular, it is important to set the SF appropriately according to the distance between the LoRa device and the gateway since the signal reception sensitivity and bit rate depend on the used SF, which are in a trade-off relationship. In addition, considering the surge in the number of LoRa devices recently, the scalability of LoRa systems is also greatly affected by the channels that the LoRa devices use for communications. It was demonstrated that the lightweight decentralized learning-based joint channel and SF-selection methods can make appropriate decisions with low computational complexity and power consumption in our previous study. However, the effect of the location situation of the LoRa devices on the communication performance in a practical larger-scale LoRa system has not been studied. Hence, to clarify the effect of the location situation of the LoRa devices on the communication performance in LoRa systems, in this paper, we implemented and evaluated the learning-based joint channel and SF-selection methods in a practical LoRa system. In the learning-based methods, the channel and SF are decided only based on the ACKnowledge information. The learning methods evaluated in this paper were the Tug of War dynamics, Upper Confidence Bound 1, and 𝜖
-greedy algorithms. Moreover, to consider the relevance of the channel and SF, we propose a combinational multi-armed bandit-based joint channel and SF-selection method. Compared with the independent methods, the combinations of the channel and SF are set as arms. Conversely, the SF and channel are set as independent arms in the independent methods that are evaluated in our previous work. From the experimental results, we can see the following points. First, the combinatorial methods can achieve a higher frame success rate and fairness than the independent methods. In addition, the FSR can be improved by joint channel and SF selection compared to SF selection only. Moreover, the channel and SF selection dependents on the location situation to a great extent.

LoRaWAN, one of Low-Power-Wide-Area (LPWA), has been deployed in many IoT applications due to its ability to communicate over long distances and low power consumption. However, the scalability and communi- cation performance of LoRaWAN is highly dependent on Spreading Factor (SF) and Channel (CH) allocation. In particular, it is important to configure SF appro- priately according to the distance of the LoRa device from the GateWay (GW) and the environment. In this paper, we implement and evaluate lightweight dis- tributed reinforcement learning-based SF selection methods. This method allows each IoT device to make appropriate parameter selections without requiring any prior information, but only utilizing ACKnowledge obtained from its own trans- missions. We then conducted real experiments in a small area indoors to verify that each LoRa device can autonomously and decentralized perform appropriate SF selection in response to the distance from the GW and SNR that varies depending on the surrounding environment. The results show that the implemented methods can select appropriate SF and achieve a better Frame Success Rate (FSR) than other lightweight approaches.

Unmanned aerial vehicles (UAVs) are becoming an integral part of numerous commercial and military applications. In many of these applications, the UAV is required to self-navigate in highly dynamic urban environments. This means that the UAV must have the ability to determine its location in an autonomous and real time manner. Existing localization techniques rely mainly on the Global Positioning System (GPS) and do not provide a reliable real time localization solution, particularly in dense urban environments. Our objective is to propose an effective alternative solution to enable the UAV to autonomously determine its location independent of the GPS and without message exchanges. We therefore propose utilizing the existing 5G cellular infrastructure to enable the UAV to determine its 3-D location without the need to interact with the cellular network. We formulate the UAV localization problem to minimize the error of the RSSI measurements from the surrounding cellular base stations. While exact optimization techniques can be applied to accurately solve such a problem, they cannot provide the real time calculation that is needed in such dynamic applications. Machine learning based techniques are strong candidates to provide an attractive alternative to provide a near-optimal localization solution with the needed practical real-time calculation. Accordingly, we propose two machine learning-based approaches, namely, deep neural network and reinforcement learning based approaches, to solve the formulated UAV localization problem in real time. We then provide a detailed comparative analysis for each of the proposed localization techniques along with a comparison with the optimization-based techniques as well as other techniques from the literature.

The fifth generation mobile networks (5G) envisions to interconnect massive numbers of devices with a wide range of characteristics and demands for 2020 and beyond. With more radio-frequency (RF) bands to support multiple frequency transmission, separate antennas and RF chips may be required for widely separated frequency bands. The development of cognitive radio systems in which a wireless transceiver automatically adapts its communication parameters to network and user demands, is therefore crucial to the successful roll-out of the 5G. This paper introduces an innovative CR system that performs spectrum sensing for joint energy harvesting and channel access. A prototype testbed is employed to detect and convert RF signals from different bands to a DC voltage for powering a sensor board that communicates over another channel to an access point. The system's objective is to use the harvested energy for concurrent data transmission. We model the selection of channels to maximize this objective as a Multi-Armed Bandit (MAB) problem. Depending on the type of the MAB problem, three spectrum sensing strategies are developed for joint energy harvesting and channel access. The first one, applicable for stochastic MAB, conceptualizes some formulations of the exploration/exploitation balancing technique of the available frequency bands. The second strategy is an opportunistic sensing framework for Markovian MAB when the state of the underlying Markov process is partially observed. The third strategy is based on the Gittins index allocation framework for fully observed Markovian MAB. The simulation results show that lower regrets can be obtained with more information on the underlying Markov process of the MAB.

The anticipated increase in the count of IoT devices in the coming years motivates the development of efficient algorithms that can help in their effective management while keeping the power consumption low. In this paper, we propose an intelligent multi-channel resource allocation algorithm for dense LoRa networks termed as LoRaDRL and provide a detailed performance evaluation. Our results demonstrate that the proposed algorithm not only significantly improves LoRaWAN's packet delivery ratio (PDR) but is also able to support mobile end-devices (EDs) while ensuring lower power consumption hence increasing both the lifetime and capacity of the network. Most previous works focus on proposing different MAC protocols for improving the network capacity, i.e., Lo-RaWAN, delay before transmit etc. We show that through the use of LoRaDRL, we can achieve the same efficiency with ALOHA compared to LoRaSim, and LoRa-MAB while moving the complexity from EDs to the gateway thus making the EDs simpler and cheaper. Furthermore, we test the performance of LoRaDRL under large-scale frequency jamming attacks and show its adaptiveness to the changes in the environment. We show that LoRaDRL's output improves the performance of state-of-the-art techniques resulting in some cases an improvement of more than 500% in terms of PDR compared to learning-based techniques.

Full-duplex relaying is an enabling technique of sixth generation (6G) mobile networks, promising tremendous rate and spectral efficiency gains. In order to improve the performance of full-duplex communications, power control is a viable way of avoiding excessive loop interference at the relay. Unfortunately, power control requires channel state information of both source-relay and relay-destination channels, as well as of the loop interference channel, thus resulting in increased overheads. Aiming to offer a low-complexity alternative for power control in full-duplex relay networks, we adopt reward-based learning in the sense of multi-armed bandits. More specifically, we provide bandit-based power control algorithms, relying on acknowledgements/negative-acknowledgements observations by the relay. The proposed algorithms avoid the need for channel state information acquisition and exchange, and can be employed in a distributed manner. Performance evaluation results in terms of outage probability, average throughput and accumulated regret over time highlight an interesting performance-complexity trade-off compared to optimal power control with full channel knowledge and significant performance gains over the cases without power control and random power level selection. Index Terms-Full-duplex relaying, power control, machine learning, multi-armed bandits.

The anticipated increase in the count of IoT devices in the coming years motivates the development of efficient algorithms that can help in their effective management while keeping the power consumption low. In this paper, we propose an intelligent multi-channel resource allocation algorithm for dense LoRa networks termed as LoRaDRL and provide a detailed performance evaluation. Our results demonstrate that the proposed algorithm not only significantly improves LoRaWAN's packet delivery ratio (PDR) but is also able to support mobile end-devices (EDs) while ensuring lower power consumption hence increasing both the lifetime and capacity of the network. Most previous works focus on proposing different MAC protocols for improving the network capacity, i.e., Lo-RaWAN, delay before transmit etc. We show that through the use of LoRaDRL, we can achieve the same efficiency with ALOHA compared to LoRaSim, and LoRa-MAB while moving the complexity from EDs to the gateway thus making the EDs simpler and cheaper. Furthermore, we test the performance of LoRaDRL under large-scale frequency jamming attacks and show its adaptiveness to the changes in the environment. We show that LoRaDRL's output improves the performance of state-of-the-art techniques resulting in some cases an improvement of more than 500% in terms of PDR compared to learning-based techniques.

Bluetooth Low Energy (BLE) has been applied to a variety of IoT applications due to the spread of Bluetooth-enabled devices and their low power consumption. However, further low power consumption can be achieved by efficient selection of advertisement intervals and channels. We propose an efficient selection method based on the Multi-Armed Bandit (MAB) algorithm, which takes the wireless environment into account for the channel and the advertising interval. In this paper, we evaluate the proposed method by simulations under various environments. The results show that the proposed method can reduce power consumption according to the wireless environment without significant loss of reliability. Furthermore, we confirmed that the proposed method works with actual BLE devices.

Spectrum under-utilization calls for the open and dynamic spectrum access mechanism, which allows the unlicensed user equipped with cognitive radios to opportunistically sense and access the spectrum that not occupied by primary users. In practice due to the hardware limitations, each cognitive radio user may be only able to sense a portion of the interested wide span spectrum. Hence, a hardware-constrained cognitive MAC to conduct efficient and intelligent spectrum sense decision is desired. In this paper, we formulate the cognitive radio spectrum sensing problem under time-varying channels as an adversarial bandit problem without any assumption of the channel statistics. A fully distributed strategy is proposed to address the fundamental tradeoff between spectrum exploration and spectrum exploitation during the sensing periods. Simulation results demonstrate that significant performance gain can be achieved by the proposed algorithm when the channels are timevarying on small timescales. A coordination scheme for the multiuser case is also presented and the effectiveness is also demonstrated by the simulation results. This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the WCNC 2010 proceedings.

Access point (AP) coordinated spatial reuse with Q-learning enables efficient spectrum utilization [1]. Although sharing of transmission schedules among APs is necessary for coordination, there is no mechanism to identify the APs with which the schedules are to be shared, resulting in excess information being shared among APs. In this study, we propose a scheme to identify the interference-free APs that are not required for sharing of information by comparing Q-values. A simple simulation demonstrates that this scheme successfully reduces shared information without throughput degradation.

The "tug-of-war (TOW) model" proposed in our previous study was a unique method for parallel searches inspired by the photoavoidance behavior of a single-celled amoeba, the true slime mold Physarum. In the TOW model, many branches of the amoeba act as search agents to collect information on light stimulations while conserving the total sum of their resources. We showed that the "nonlocal correlation" via resource conservation can be advantageous to manage the "exploration-exploitation dilemma" for solving the multi-armed bandit problem. In this study, toward a physical implementation of the TOW model, we propose a simpler model called "solid TOW model" which uses the origin of high performance of the TOW model more directly, and evaluate the model's performance.

従来のコンピュータでは、巡回セールスマン問題(Traveling Salesman Problem)に代表されるよう な、組み合わせ最適化問題を効率的に解くことが困難である。しかし、この問題を解くことは、 人工知能の開発や大規模で複雑化した社会システムの最適化への応用に向け、必要不可欠である。 近年、組み合わせ最適化問題を解くために、イジングスピンモデルを磁束量子ビット[1]や半導体 CMOS回路[2]を用いてハードウェアに実装したものが報告され、注目を集めている。イジングス ピンモデルとは、格子状に配置した強磁性体のスピン系であり(図 1)、この系のハミルトニアン H は、スピン間の相互作用の強さを表す J、スピンの向きを表す σ、磁場を表す h を用いて、 と記述される。この系のエネルギーの自然収束動作が組み合わせ最適化問 題へと写像できることが知られている。そこで、我々は、イジングスピンモデルをプログラマブ ルロジックデバイスである FPGA(Field-Programmable Gate Array)に実装し、自然現象を模擬した新 しい計算技術の検討を行った。 今回、イジングスピンモデルを FPGAに実装し、組み合わせ最適化問題のひとつである「最大カ ット問題」への適応を検討した。はじめに、2次元イジングスピンモデル(10×10)に対して、系が基 底状態のときに、アルファベットの「M」が浮かび上がるように初期条件を設定し、計算を実行し た。図 2 (a) (c)に時間に対するスピン状態の変化を示す。これより、時間の経過とともに「M」と いう文字が徐々に浮かび上がる様子が確認された。また、このスピン系は 2 の状態を有してい るにも関わらず、58 msec で最適解へと導くことができた。以上のことから、FPGA を用いてハー ドウェアに実装したイジングスピンモデルにより、組み合わせ最適化問題の取り扱いが可能であ ると考えられる。

We focus on the budget-limited multi-armed bandit (BL-MAB) problems. In BL-MAB problems, the agent's actions are costly and constrained by a fixed budget. The existing BL-MAB problems assume the reward distributions are static. We assume the reward distributions are dynamic. It is more natural to assume the reward distributions are dynamic. For example, online advertisement's effect is dynamic for the trends and dates. Online advertisement is one of the real-world applications of BL-MAB problems. We make new bandit algorithms D-KUBE and SW-KUBE for dynamic situations. In our experiments, we compared the existing bandit algorithm with our proposed bandit algorithms. Experiment results show that D-KUBE and SW-KUBE are better than KUBE for the dynamic reward distributions.

Structural health monitoring attracts attention, as the technique of realizing quantification of the damage risk of building and civil structure. However, the cost spent on wiring had become the barrier which introduces a monitoring system to a large-sized structure. In this research, for wiring cost reduction, the wireless sensor network system which used TCP/IP and radio communications was proposed, and the theoretical analysis, prototype manufacture, and the evaluation examination were performed. The sensor network proposed in this paper satisfied the target performance. And it was able to reduce sharply the wiring cost and the time. Furthermore, it has attained the high synchronicity needed for real time damage detection.

In recent years, the demand for new applications using various Internet of Things (IoT) devices has led to an increase in the number of devices connected to wireless networks. However, owing to the limitation of available frequency resources for IoT devices, the degradation of the communication quality caused by channel congestion is a practical problem in developing IoT technology. Many IoT devices have hardware and software limitations that prevent centralized channel allocation, and congestion is even more severe in massive IoT networks without a central controller. Therefore, developing a distributed and sophisticated channel selection algorithm is necessary. In previous studies, the channel selection of each IoT device was modeled as a multi-armed bandit (MAB) problem, and a wireless channel selection method based on the MAB algorithm, which is a simple reinforcement learning, was proposed. In particular, it has been shown that the MAB algorithm of tug-of-war (TOW) dynamics can efficiently select channels with much lower computational complexity and power compared with other reinforcement learning-based channel- selection methods. This paper proposes a distributed channel selection method based on TOW dynamics in fully decentralized networks. We evaluate the effectiveness of the proposed method and other distributed channel-selection methods on the communication success rate in massive IoT networks by experiments and simulations. The results show that the proposed method improves the communication success rate more than other distributed channel selection methods even in a dense and dynamic network environment.

In recent times, the number of Internet of Things devices has increased considerably. Numerous Internet of Things devices generate enormous traffic, thereby causing network congestion and packet loss. To address network congestion in massive Internet of Things systems, an efficient channel allocation method is necessary. Although some channel allocation methods have already been studied, as far as we know, there is no research focusing on the implementation phase of Inter- net of Things devices while considering massive heterogeneous Internet of Things systems where different kinds of Internet of Things devices coexist in the same Internet of Things system. This paper focuses on the multi-armed-bandit-based channel allocation method that can be implemented on resource-constrained Internet of Things devices with low computational processing ability while avoiding congestion in massive Internet of Things systems. This paper first evaluates some well- known multi-armed-bandit-based channel allocation methods in massive Internet of Things systems. The simulation results show that an improved multi-armed-bandit-based channel selection method called Modified Tug of War can achieve the highest frame success rate in most cases. Specifically, the frame success rate can reach 95% when the numbers of channels and IoT devices are 60 and 10,000, respectively, while 12% channels are suffering traffic load by other kinds of IoT devices. In addition, the performance in terms of frame success rate can be improved by 20% compared to the equality channel allocation. Moreover, the multi-armed-bandit-based channel allocation methods is implemented on 50 Wi-SUN Internet of Things devices that support IEEE 802.15.4g/4e commu- nication and evaluate the performance in frame success rate in an actual wood house coexisting with LoRa devices. The experimental results show that the modified multi-armed-bandit method can achieve the highest frame success rate compared to other well-known frame success rate-based channel selection methods.

Spectrum under-utilization calls for the open and dynamic spectrum access mechanism, which allows the unlicensed user equipped with cognitive radios to opportunistically sense and access the spectrum that not occupied by primary users. In practice due to the hardware limitations, each cognitive radio user may be only able to sense a portion of the interested wide span spectrum. Hence, a hardware-constrained cognitive MAC to conduct efficient and intelligent spectrum sense decision is desired. In this paper, we formulate the cognitive radio spectrum sensing problem under time-varying channels as an adversarial bandit problem without any assumption of the channel statistics. A fully distributed strategy is proposed to address the fundamental tradeoff between spectrum exploration and spectrum exploitation during the sensing periods. Simulation results demonstrate that significant performance gain can be achieved by the proposed algorithm when the channels are timevarying on small timescales. A coordination scheme for the multiuser case is also presented and the effectiveness is also demonstrated by the simulation results. This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the WCNC 2010 proceedings.

In recent times, the number of Internet of Things devices has increased considerably. Numerous Internet of Things devices generate enormous traffic, thereby causing network congestion and packet loss. To address network congestion in massive Internet of Things systems, an efficient channel allocation method is necessary. Although some channel allocation methods have already been studied, as far as we know, there is no research focusing on the implementation phase of Internet of Things devices while considering massive heterogeneous Internet of Things systems where different kinds of Internet of Things devices coexist in the same Internet of Things system. This paper focuses on the multi-armed-bandit-based channel allocation method that can be implemented on resource-constrained Internet of Things devices with low computational processing ability while avoiding congestion in massive Internet of Things systems. This paper first evaluates some well-known multi-armed-bandit-based channe...

Reinforcement learning involves decision-making in dynamic and uncertain environments and constitutes a crucial element of artificial intelligence. In our previous work, we experimentally demonstrated that the ultrafast chaotic oscillatory dynamics of lasers can be used to efficiently solve the two-armed bandit problem, which requires decision-making concerning a class of difficult trade-offs called the exploration-exploitation dilemma. However, only two selections were employed in that research; hence, the scalability of the laser-chaos-based reinforcement learning should be clarified. In this study, we demonstrated a scalable, pipelined principle of resolving the multi-armed bandit problem by introducing time-division multiplexing of chaotically oscillated ultrafast time series. The experimental demonstrations in which bandit problems with up to 64 arms were successfully solved are presented where laser chaos time series significantly outperforms quasiperiodic signals, computer-ge...

Distributed power control by beamforming approach in cognitive radio networks requires a precise analysis of the impacts of the transmission parameters, tolerable interference and guarantees the quality of service of both the primary users and secondary users. In this paper, we propose an improved performance to solve the constrained nonlinear multi-object optimization and coherent power assignment by beamforming problem based on particle swarm optimization. This method is invoked to solve the constrained nonlinear optimization problem in order to reach to maximum capacity. A numerical study is performed to show the convergence behavior of the proposed algorithm and the efficiency of the proposed technique with a dynamic cost function.

Structural health monitoring attracts attention, as the technique of realizing quantification of the damage risk of building and civil structure. However, the cost spent on wiring had become the barrier which introduces a monitoring system to a large-sized structure. In this research, for wiring cost reduction, the wireless sensor network system which used TCP/IP and radio communications was proposed, and the theoretical analysis, prototype manufacture, and the evaluation examination were performed. The sensor network proposed in this paper satisfied the target performance. And it was able to reduce sharply the wiring cost and the time. Furthermore, it has attained the high synchronicity needed for real time damage detection.

For Massive IoT systems, various Low Power Wide Area (LPWA) systems have been developed and deployed, i.e., LoRa, SigFox, etc. In this paper, to avoid destructive collisions when multiple IoT LoRa signals simultaneously received in the same channel, we propose a Successive Interference Cancellation (SIC) based collision avoidance mechanism by accessing channel using reinforcement learning for distributed massive IoT systems. Simulation results show the effectiveness of our proposed mechanism in terms of Frame Success Rate (FSR).

The petroleum sector has undoubtedly contributed substantially to Nigerian economy since its discovery in 1956 and more especially, since 1970 when its price has been on the upward   trend. It is important to note that for a country to attain the growth and development level which Nigeria requires; its economy has to be diversified. With a fall in the price of oil in the international market, it has become very evident to Nigeria that the only option left to sustain her economy is looking outwards and looking away from oil as the only source of improving its GDP. Nigeria is a mono product nation and this mono productivism has put the nation in an economic quagmire. A strong growing sustain able economy is the goal of every nation. Mono product economy needs to give way to the productive development of various sectors of the economy. The study f ocused on economic diversification and the role of agricultural export therein. Most studies have concentrated on diversification of agricul...

For Massive IoT systems, various Low Power Wide Area (LPWA) systems have been developed and deployed, i.e., LoRa, SigFox, etc. In this paper, to avoid destructive collisions when multiple IoT LoRa signals simultaneously received in the same channel, we propose a Successive Interference Cancellation (SIC) based collision avoidance mechanism by accessing channel using reinforcement learning for distributed massive IoT systems. Simulation results show the effectiveness of our proposed mechanism in terms of Frame Success Rate (FSR).

High-bandwidth irregular oscillations caused by optical time-delayed feedback subjected to the laser cavity, known as laser chaos, have been investigated for various engineering applications. Recently, a fast decision-making algorithm for a multi-arm bandit problem by utilizing laser chaos time series has been demonstrated. Furthermore, the arms order recognition of the reward expectation for each arm has been successfully developed by incorporating the notion of the confidence interval regarding the reward estimate. However, in previous studies, the verification was limited to numerical experiments; real-world demonstrations were not conducted. This study experimentally demonstrated that the arm-order recognition algorithm is successfully operated in channel order recognition in wireless communications while revising the original strategy to take into account the wireless application requirements. Such accurate arm rank recognition involving non-best arms would be useful for various real-world applications such as channel bonding, among others.

The wide spread of mobile communication devices has increased the opportunities to use wireless communication technologies, irrespective of one's geographical location. However communication quality deteriorates due to factors such as competition for scarce radio resources and interference among nearby devices. Cognitive radio technologies have been developed recently to conquer such difficulties. In this paper, we propose a wireless network optimization method using learning algorithms based on a control model known as cognitive cycle. We implement the proposed optimization method in wireless LANs and evaluate the throughput performance. The experimental results show the effectiveness of the proposed approach in a real environment.

It is stated that the women have more favourable attitude about environmental subjects according to some researches. The reason of this tendency is explained by women' EQ. The purpose of this research is to compare women' natural intelligence and environmental attitudes. behalf of EQ. As a result, there is significant differences between total scores of naturalist intelligence in favor of women while no difference between gender on attitude scale scores. The relationship between intelligence and attitude areas is assessed by regression analysis, and it is stated that natural intelligence is significant predictors in determining attitude.

This study provides a solution for mm-wave FMCW radar-based heart-rate detection of an indoor moving person. The range-bin where the person is located will change during the moving state, and the signal energy is weaker than clutter at farther distances. Therefore, we propose an adaptive range-bin selection method that effectively selects the optimal range-bin cell to obtain the correct radar vital sign before heartbeat signal reconstruction. The experimental results show that the proposed method can reconstruct the heartbeat signal with high precision when used in conjunction with the ICEEMDAN method.

Now, Virtualization Technology which attracted attention came to be used from the view- point of cost cut. It is thought that it is used still more in future by a set of processor enhancements that improve traditional software-based virtualization solutions. In particular, software called Virtual Machine Monitor (VMM or Hypervisor) attracts attention at a point of performance and the security.

The control of a group of elevators is a difficult stochastic control problem, because of the random and unpredictable passenger arrivals. Here we propose a new method for constructing an adaptive controller for stochastic systems, by using a combination of neural networks and an on-line reinforcement learning based on stochastic approximation. This method combines an efficient supervised learning with a general reinforcement adaptation. The supervised learning is used to prepare the controller with domain-specific knowledge, and for initializing it to emulate an existing controller. The reinforcement learning is used for adaptation, with a special attention paid to allow online operation. The new method is used to develop an adaptive, optimal elevator group controller. Results of simulation tests indicate the feasibility of using the proposed method for industrial applications.