CN102420867A - Cluster storage entry resolution method based on real-time load balancing mechanism - Google Patents

Abstract

The invention provides a cluster storage entry resolution method based on a real-time load balancing mechanism. System load monitoring modules are set in all interface node servers of a distributed file system, the real-time system loads of all the interface node servers are acquired by a system accessing entry server, and service allocation Internet protocol (IP) addresses born by interface nodes with relatively 'lower' system loads are allocated to new access requests of the distributed file system so as to balance access requests of the distributed file system in real time in an interface node server cluster.

Description

A cluster storage entry analysis method based on real-time load balancing mechanism

technical field

The present invention relates to a cloud computer application technology, in particular to a cluster storage entry analysis method based on a real-time load balancing mechanism.

Background technique

With the rapid development and application of cloud computing and cloud storage, more and more distributed file systems are deployed in the storage backend. At present, the access request entry analysis method for the distributed file system namespace mainly adopts fixed IP address allocation or DNS-based Round Robin IP queue allocation. These two methods do not pay attention to the real-time load of the interface node server, nor do they pay attention to the interface server. The cluster balances access requests, especially when the access load pressure of the interface node server is already overwhelmed, it still needs to continue to undertake external access request tasks, which eventually makes the system abnormal until it crashes. As cloud computing and cloud storage applications have higher and higher requirements on the capacity and performance of storage systems, large-capacity storage systems have gradually developed from expansion methods of disks and arrays to distributed storage methods. As the core of horizontally scalable storage systems, Distributed file system has been greatly developed and applied.

The distributed file system provides storage space to the interface node server in the form of providing a unified namespace of the file system, and the interface node server undertakes read and write access requests for the distributed file system from outside the system. In the cloud business environment, problems such as the emergence of explosive access requests in a specific period of time, the aggregation of massive access requests on a single interface node server, and the guarantee of key business access requests in the process of multi-service request access will affect the reliability of the upper-layer business applications of the system. , Consistency, so that the consistency of the file system data.

Contents of the invention

The purpose of the present invention is to provide a cluster storage entry analysis method based on a real-time load balancing mechanism.

The purpose of the present invention is achieved in the following manner, add a system load monitoring module to all interface node servers of the distributed file system, and obtain the real-time system load of all interface node servers through the system access entry server, and make the system load relatively "idle" The service allocation IP assigned by the interface node is assigned to the new access request of the distributed file system, and the access request of the distributed file system can be balanced in real time on the server cluster of the interface node. The specific steps are as follows:

1) By setting the system load monitoring module on the interface node server, accessing the intelligent load evaluation module of the entry server through the distributed file system to obtain the interface node server and service IP with a relatively "leisure" load, the access request entry analysis module will provide the load evaluation module with The service IP assigned to the access request of the system realizes the assignment of the access request to the service IP of the most "leisure" interface node server;

2) The system load monitoring module deployed on the interface node server realizes a more timely and accurate status through the real-time monitoring of the access tasks carried by the interface node server and the system hardware load, as well as the judgment that the interface node server can continue to bear the load Monitoring; if there is a control node server in the distributed file system, a monitoring module is deployed on the control node server to monitor the load of the control node server;

3) Set up an intelligent load evaluation module on the distributed file system access entry server. This module is a comprehensive evaluation module based on the monitored data of the system. The system load monitoring module is deployed on a single interface node server. Monitor the system load, make statistics on the related information of the access requests undertaken, and the maximum bearable load of the interface node server, and the intelligent load evaluation module is to carry out the related data of all the interface node servers that have borne the load and can continue to bear the access load. Statistics, evaluate and calculate according to the interface node server load evaluation strategy set by the system, and generate the sustainable load factor of the interface node server and external service IP in the system;

4) Set the parsing module at the access request entry of the distributed file system access entry server. This module is the interface module of the system for external access requests. For access requests from outside the system, it calls the interface node server generated by the intelligent load evaluation module and external The service IP queue can continue to bear the load factor, and select the relatively "idle" service IP deployed on the "idle" interface node server to map to the access request of the system; this access request will directly communicate with the assigned service IP in the future until the request At the end, when the interface node server output by the intelligent load evaluation module or the external service IP can continue to bear the load factor exceeding the system setting value, it is set whether to refuse to provide service IP for external access requests to avoid excessive system access load.

The system load monitoring module on the interface node server detects the maximum load parameters that the interface node server can carry, obtains the access load parameters that the interface node server has undertaken in real time, and records the historical access load parameters of the interface node server. The generation of related factors set by the system can continue Take the load parameter.

The intelligent load evaluation module deployed on the system access entry server obtains the performance detection data generated by the system load monitoring module on all interface node servers, the monitoring data of the load that has been borne, and the load data that can continue to be borne, and through the set interface node server load The evaluation strategy generates the sustainable load factors of all interface node servers and the sustainable load factors of all service IPs deployed on the interface node servers.

The access request entry parsing module of the system access entry server receives the access request of the distributed file system, and obtains the interface node server with a load factor that can continue to be borne by the intelligent load evaluation module and the external service IP, according to the load factor set in the system Threshold, to determine whether to map the "idle" interface node server and "idle" service IP to the external access request of the system.

The beneficial effects of the present invention are: the present invention is applied to a distributed file system access request entry parsing system in the fields of cloud computing and cloud storage. In the application environment of various cloud services such as public cloud, industry cloud, enterprise cloud, and private cloud, its back-end storage will increasingly rely on large-capacity storage systems, among which distributed file systems based on distributed The storage system has gradually become the mainstream application of cloud services. It is used to establish the management server cluster, data server cluster, and interface node server cluster of the distributed file system, and set the corresponding monitoring parameters of the monitoring module in the interface node server and the corresponding monitoring parameters of the monitoring module in the management server (if it exists). It is also used to establish the access request entry server of the distributed file system, set the information source of the intelligent evaluation module and the entry analysis module: monitoring interface node server information, external service IP queue, access request source range, load threshold and range, etc.

Description of drawings

Figure 1 Deployment structure of interface node server monitoring module

Figure 2 Flow chart of the intelligent load evaluation module;

Figure 3 Flowchart of the entry parsing module.

Detailed ways

With reference to accompanying drawing of description, method of the present invention is described in detail below.

In the process of providing massive storage capacity in the distributed file system, the access requests of the global namespace inevitably have access control, access session management, balanced distribution of access requests, guarantee of access requests based on specific services, and handling of abnormal access requests, etc. The emergence of these problems will affect the operating efficiency of the distributed file system namespace, the operating efficiency of access requests and upper-layer services, the guarantee of data consistency, and the session congestion of access requests.

In view of this, the present invention proposes to introduce a monitoring module into the interface node server (including the control node server) of the distributed file system, so as to monitor the access load of the distributed file system, the operating load of the interface node, and the interface node server cluster in the interface node server cluster. The maximum performance parameters of the interface node server; monitoring the maximum performance parameters of the interface node server can know the maximum performance load that the node can carry, and can provide more considerable performance data to the evaluation module through the change rate of the load that has been borne and the load that can continue to be borne. The intelligent load evaluation module is introduced into the access entrance server to realize the evaluation of the load already borne by the interface node server cluster and the external service IP, and the load that can continue to be borne; the intelligent load evaluation module calculates and analyzes the data of the monitoring module, and outputs the interface node server and The sustainable load factor of the external service IP is used as the basis for the request entry analysis module to select the mapping IP. The access request entry analysis module is introduced into the access entry server to realize the analysis of external access requests, the selection and mapping of the interface node server and the external service IP; The output node server with the interface that can continue to bear the load factor and the external service IP realize the operation of the external access request entry analysis system of the unified namespace of the distributed file system.

The method of the present invention includes the following three parts: a system load monitoring module on the interface node server, an intelligent load evaluation module on the access entry server, and an access request entry analysis module on the access entry server, wherein:

The system load monitoring module on the interface node server is a system load information monitoring and collection module deployed on the interface node server, which is used to obtain the maximum operating performance information (such as CPU, memory, network, etc.), actual load, and assumed load of the interface node system. The rate of change of load information, etc., is an important basis for evaluating whether the interface node can continue to bear the load.

The intelligent load evaluation module on the access entry server is a module deployed on the access entry server to evaluate the load factor that the interface node server cluster and the external service IP queue can continue to bear. According to the operating load data collected by the monitoring module, the interface node server generates The "idle degree" evaluation of the interface node server and external service IP provides an evaluation basis for realizing the balanced distribution of real-time load of access requests in the interface node server cluster.

The access request entry parsing module of the access entry server is the first processing module deployed on the access entry server to receive the external access request of the system, and is used to judge the source of the external request; determine the external service on the most "idle" interface node according to the source IP mapping; when the system access pressure exceeds the maximum threshold, the mapping of external service IP is suspended; when the load pressure of the interface node server exceeds the threshold, the assignment of access request tasks to it is suspended. This module is the entry module to realize the distributed file system, and also the main control module to realize the real-time dynamic load balance of the interface node server.

Except for the technical features described in the instructions, all are known technologies by those skilled in the art.

Claims (4)

1. A cluster storage entry analysis method based on a real-time load balancing mechanism, which is characterized in that a system load monitoring module is added to all interface node servers of the distributed file system, and the real-time system load of all interface node servers is obtained through system access to the entry server , allocate the service allocation IP undertaken by the interface node with a relatively "idle" system load to the newly added access request of the distributed file system, and realize the real-time balanced hosting of the access request of the distributed file system in the interface node server cluster. The specific steps are as follows: 1) By setting the system load monitoring module on the interface node server, accessing the intelligent load evaluation module of the entry server through the distributed file system to obtain the interface node server and service IP with a relatively "leisure" load, the access request entry analysis module will provide the load evaluation module with The service IP assigned to the access request of the system realizes the assignment of the access request to the service IP of the most "leisure" interface node server; 2) The system load monitoring module deployed on the interface node server realizes a more timely and accurate status through the real-time monitoring of the access tasks carried by the interface node server and the system hardware load, as well as the judgment that the interface node server can continue to bear the load Monitoring; if there is a control node server in the distributed file system, a monitoring module is deployed on the control node server to monitor the load of the control node server; 3) Set up an intelligent load evaluation module on the distributed file system access entry server. This module is a comprehensive evaluation module based on the monitored data of the system. The system load monitoring module is deployed on a single interface node server. Monitor the system load, make statistics on the related information of the access requests undertaken, and the maximum bearable load of the interface node server, and the intelligent load evaluation module is to carry out the related data of all the interface node servers that have borne the load and can continue to bear the access load. Statistics, evaluate and calculate according to the interface node server load evaluation strategy set by the system, and generate the sustainable load factor of the interface node server and external service IP in the system; 4) Set the parsing module at the access request entry of the distributed file system access entry server. This module is the interface module of the system for external access requests. For access requests from outside the system, it calls the interface node server generated by the intelligent load evaluation module and external The service IP queue can continue to bear the load factor, and select the relatively "idle" service IP deployed on the "idle" interface node server to map to the access request of the system; this access request will directly communicate with the assigned service IP in the future until the request At the end, when the interface node server output by the intelligent load evaluation module or the external service IP can continue to bear the load factor exceeding the system setting value, it is set whether to refuse to provide service IP for external access requests to avoid excessive system access load. 2. The method according to claim 1, wherein the system load monitoring module on the interface node server detects the maximum load parameter that the interface node server can carry, obtains the interface node server in real time and has undertaken the access load parameter and records the interface node server The historical access load parameters can be generated through the correlation factors set by the system to continue to bear the load parameters. 3. The method according to claim 1, wherein the intelligent load evaluation module deployed on the system access entry server obtains the performance detection data generated by the system load monitoring module on all interface node servers, the monitoring data of the assumed load, and The load data that can continue to be borne, through the set interface node server load evaluation strategy, generates the sustainable load factor of all interface node servers and the sustainable load factor of all service IPs deployed on the interface node server. 4. The method according to claim 1, characterized in that, the access request entry parsing module of the system access entry server receives the access request of the distributed file system, and obtains the interface output by the intelligent load evaluation module that can continue to bear the load factor The node server and the external service IP, according to the load factor threshold set in the system, determine whether to map the "idle" interface node server and the "idle" service IP to the external access request of the system.

Images