Overmind: A Collaborative Decentralized Machine Learning Framework

J. Inf. Data Manag., 2018

Climate prediction is a relevant activity for humanity and, for the success of the climate forecast, a good historical database is necessary. However, because of several factors, large historical data gaps are found at different meteorological stations, and studies to determine such missing weather values are still scarce. This work describes a study of a combination of several machine learning techniques to determine missing climatic values. This study extends our previous work, producing a computational framework, formed by three different methods: neural networks, regression bagged trees and random forest. Deep data analysis and a statistical study is conducted to compare these three methods. The study statistically demonstrated that the random forest technique was successful in obtaining missing climatic values for the state of Minas Gerais and can be widely used by the responsible agencies to improve their historical databases, consequently, their climate forecasts.

2022

The agricultural sector's day-today operations, such as irrigation and sowing, are impacted by the weather. Therefore, weather constitutes a key role in all regular human activities. Weather forecasting must be accurate and precise to plan our activities and safeguard ourselves as well as our property from disasters. Rainfall, wind speed, humidity, wind direction, cloud, temperature, and other weather forecasting variables are used in this work for weather prediction. Many research works have been conducted on weather forecasting. The drawbacks of existing approaches are that they are less effective, inaccurate, and time-consuming. To overcome these issues, this paper proposes an enhanced and reliable weather forecasting technique. As well as developing weather forecasting in remote areas. Weather data analysis and machine learning techniques, such as Gradient Boosting Decision Tree, Random Forest, Naive Bayes Bernoulli, and KNN Algorithm are deployed to anticipate weather conditions. A comparative analysis of result outcome said in determining the number of ensemble methods that may be utilized to improve the accuracy of prediction in weather forecasting. The aim of this study is to demonstrate its ability to predict weather forecasts as soon as possible. Experimental evaluation shows our ensemble technique achieves 95% prediction accuracy. Also, for 1000 nodes it is less than 10 s for prediction, and for 5000 nodes it takes less than 40 s for prediction.

2024

The rapid development of modern information technology (IT), power supply, communication and traffic information systems and so on is resulting in progress in the area of distributed and energy-efficient (if possible, powered by renewable energy sources) smart grid components securely connected to entire smart city management systems. This enables a wide range of applications such as distributed energy management, system health forecasting and cybersecurity based on huge volumes of data that automate and improve the performance of the smart grid, but also require analysis, inference and prediction using artificial intelligence. Data management strategies, but also the sharing of data by consumers, institutions, organisations and industries, can be supported by edge clouds, thus protecting privacy and improving performance. This article presents and develops the authors’ own concept in this area, which is planned for research in the coming years. The paper aims to develop and initially test a conceptual framework that takes into account the aspects discussed above, emphasising the practical aspects and use cases of the Social Internet of Things (SIoT) and artificial intelligence (AI) in the everyday lives of smart sustainable city (SSC) residents. We present an approach consisting of seven algorithms for the integration of large data sets for machine learning processing to be applied in optimisation in the context of smart cities.

2020 54th Annual Conference on Information Sciences and Systems (CISS)

In this paper, we propose a fast, privacy-aware, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm is based on the Alternating Direction Method of Multipliers (ADMM) algorithm. The key novelty in the proposed algorithm is that it solves the problem in a decentralized topology where at most half of the workers are competing the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges faster than the centralized batch gradient descent for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets.

international journal of engineering technology and management sciences

Federated learning (also known as collaborative learning) is a machine learning technique that trains an algorithm without transferring data samples across numerous decentralized edge devices or servers. This strategy differs from standard centralized machine learning techniques in which all local datasets are uploaded to a single server, as well as more traditional decentralized alternatives, which frequently presume that local data samples are uniformly distributed. Federated learning allows several actors to collaborate on the development of a single, robust machine learning model without sharing data, allowing crucial issues such as data privacy, data security, data access rights, and access to heterogeneous data to be addressed. Defence, telecommunications, internet of things, and pharmaceutical industries are just a few of the sectors where it has applications.

Journal of Engineering Design and Technology, 2023

Purpose The study aims to develop a multilayer high-effective ensemble of ensembles predictive model (stacking ensemble) using several hyperparameter optimized ensemble machine learning (ML) methods (bagging and boosting ensembles) trained with high-volume data points retrieved from Internet of Things (IoT) emission sensors, time-corresponding meteorology and traffic data. Design/methodology/approach For a start, the study experimented big data hypothesis theory by developing sample ensemble predictive models on different data sample sizes and compared their results. Second, it developed a standalone model and several bagging and boosting ensemble models and compared their results. Finally, it used the best performing bagging and boosting predictive models as input estimators to develop a novel multilayer high-effective stacking ensemble predictive model. Findings Results proved data size to be one of the main determinants to ensemble ML predictive power. Second, it proved that, as compared to using a single algorithm, the cumulative result from ensemble ML algorithms is usually always better in terms of predicted accuracy. Finally, it proved stacking ensemble to be a better model for predicting PM 2.5 concentration level than bagging and boosting ensemble models. Research limitations/implications A limitation of this study is the trade-off between performance of this novel model and the computational time required to train it. Whether this gap can be closed remains an open research question. As a result, future research should attempt to close this gap. Also, future studies can integrate this novel model to a personal air quality messaging system to inform public of pollution levels and improve public access to air quality forecast. Practical implications The outcome of this study will aid the public to proactively identify highly polluted areas thus potentially reducing pollution-associated/ triggered COVID-19 (and other lung diseases) deaths/ complications/ transmission by encouraging avoidance behavior and support informed decision to lock down by government bodies when integrated into an air pollution monitoring system Originality/value This study fills a gap in literature by providing a justification for selecting appropriate ensemble ML algorithms for PM 2.5 concentration level predictive modeling. Second, it contributes to the big data hypothesis theory, which suggests that data size is one of the most important factors of ML predictive capability. Third, it supports the premise that when using ensemble ML algorithms, the cumulative output is usually always better in terms of predicted accuracy than using a single algorithm. Finally developing a novel multilayer high-performant hyperparameter optimized ensemble of ensembles predictive model that can accurately predict PM 2.5 concentration levels with improved model interpretability and enhanced generalizability, as well as the provision of a novel databank of historic pollution data from IoT emission sensors that can be purchased for research, consultancy and policymaking.

Sensors

The sparse data in PM2.5 air quality monitoring systems is frequently happened on large-scale smart city sensing applications, which is collected via massive sensors. Moreover, it could be affected by inefficient node deployment, insufficient communication, and fragmented records, which is the main challenge of the high-resolution prediction system. In addition, data privacy in the existing centralized air quality prediction system cannot be ensured because the data which are mined from end sensory nodes constantly exposed to the network. Therefore, this paper proposes a novel edge computing framework, named Federated Compressed Learning (FCL), which provides efficient data generation while ensuring data privacy for PM2.5 predictions in the application of smart city sensing. The proposed scheme inherits the basic ideas of the compression technique, regional joint learning, and considers a secure data exchange. Thus, it could reduce the data quantity while preserving data privacy. Th...

Procedia Computer Science, 2019

The problem of the weather forecasting still exists in urban agglomerations, in part because of local factors on a city scale. It is important to take into acount urban processes, to make a high-resolution weather forecast. In this study, we suggest an approach to use crowdsourced meteorological data from citizen weather stations (CWS) to improve weather forecasting. The Weather Research and Forecasting model was used to simulate forecasting for two cities Saint Petersburg and Moscow. Using processed CWS data we propose a data-driven model to correct numeric modeling for short-time forecasting. We propose a procedure for statistical validation of citizen stations using observations from professional meteostations. Experiments have shown that in quality they can be compared with the government ones, the numerical model WRF need to be corrected for representing winter heat island. We considered the possibility of using text data from social media posts to predict weather conditions in the case of air temperature.

The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020

Air pollution, which is one of the biggest problems created by the developing world, reaches severe levels, especially in urban areas. Weather stations established at certain points in countries regularly obtain data and inform people about air quality. In Smart City applications, it is aimed to perform this process with higher speed and accuracy by collecting data with thousands of sensors based on the Internet of Things. At this stage, artificial intelligence and machine learning plays a vital role in analyzing the data to be obtained. In this study, six pollutant concentrations; particulate matters (PM2.5 and PM10), nitrogen dioxide (NO2), sulfur dioxide (SO2), Ozone (O3), and carbon monoxide (CO), were predicted using three basic machine learning algorithms, namely, random forest, decision tree and support vector regression, by considering only meteorological data. Experiments on two different datasets showed that the random forest has a high prediction capacity (R 2 :0.74-0.86), and high-accuracy predictions can be performed on pollutant concentrations using only meteorological data. This and further studies based on meteorological data would help to reduce the number of devices in Smart City applications and will make it more cost-effective.

Cornell University - arXiv, 2022

Federated Learning is a machine learning paradigm where we aim to train machine learning models in a distributed fashion. Large number of clients/edge devices collaborate with each other to train a single model on the central. Clients do not share their own dataset with each other, decoupling computation and data on the same device. In this paper, we propose yet another adaptive federated optimization method, and some other ideas in the field of federated learning. We also perform experiments using these methods and showcase the improvement in overall performance of federated learning.