CN111340220A - Method and apparatus for training predictive models - Google Patents

Abstract

The present disclosure relates to the field of artificial intelligence. Embodiments of the present disclosure disclose methods and apparatus for training a predictive model. The prediction model is used for predicting the performance of the neural network structure, and the method comprises training the prediction model through a sampling operation; the sampling operation comprises the following steps: sampling a sub-network from the trained super-network, and training the sampled sub-network to obtain the performance information of the trained sub-network; constructing sample data based on the trained sub-network and the corresponding performance information, and training a prediction model by using the sample data; and in response to determining that the precision of the prediction model trained in the current sampling operation does not meet the preset condition, executing the next sampling operation, and increasing the number of sub-networks for sampling in the next sampling operation. The method can reduce the searching cost of the neural network model structure.

Description

Method and apparatus for training predictive models

technical field

The embodiments of the present disclosure relate to the field of computer technology, in particular to the field of artificial intelligence technology, and in particular, to a method and apparatus for training a prediction model.

Background technique

With the development of artificial intelligence technology and data storage technology, deep neural networks have achieved important results in many fields. The design of the deep neural network structure has a direct impact on its performance. The design of the traditional deep neural network structure is done manually based on experience. The manual design of the network structure requires a lot of expert knowledge, and the network structure needs to be designed for different tasks or application scenarios, and the cost is high.

NAS (neural architecture search, automatic neural network structure search) is to replace tedious manual operations with algorithms to automatically search for the best neural network architecture. The existing model structure automatic search can only be searched based on specific constraints, such as searching for a specified hardware device model. However, the constraints in actual scenarios are complex and vary a lot, involving multiple types of hardware, such as multiple types of processors. For each type of hardware, search constraints are also numerous, such as different delay constraints. Existing methods need to perform network structure search for each constraint condition, and a large number of repetitive network structure search tasks will consume a lot of computing resources and the cost is very high.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure propose methods and apparatuses, electronic devices, and computer-readable media for training predictive models.

In a first aspect, embodiments of the present disclosure provide a method for training a prediction model, where the prediction model is used to predict the performance of a neural network structure, and the method for training a prediction model includes training the prediction model through a sampling operation; the sampling operation includes: : Sample the sub-network from the trained super-network, and train the sampled sub-network to obtain the performance information of the trained sub-network; build the sample data based on the trained sub-network and the corresponding performance information, and Use the sample data to train the prediction model; in response to determining that the accuracy of the prediction model trained in the current sampling operation does not meet the preset condition, perform the next sampling operation, and increase the number of sampled sub-networks in the next sampling operation.

In some embodiments, the above-mentioned sampling of the sub-network from the trained super-network includes: sampling the sub-network from the trained super-network by using an initial recurrent neural network; Before training, the sampling operation also includes: generating feedback information based on the performance information of the trained sub-network to iteratively update the recurrent neural network based on the feedback information; re-sampling from the trained super-network based on the iteratively updated recurrent neural network subnet.

In some embodiments, the above-mentioned sampling of the sub-network from the trained super-network includes: sampling the unsampled sub-network from the trained super-network; and the above-mentioned based on the trained sub-network and corresponding The construction of sample data includes: constructing sample data based on the sub-network and corresponding performance information sampled in the current sampling operation, and the sub-network and corresponding performance information sampled in the previous sampling operation.

In some embodiments, the above sampling operation further includes: in response to determining that the accuracy of the prediction model meets a preset condition, generating a trained prediction model based on the training result of the current sampling operation.

In some embodiments, the above method further includes: based on the performance prediction result of the model structure in the preset model structure search space based on the trained prediction model, and the preset performance constraints of the deep learning task scenario, in the model structure The neural network model structure that satisfies the performance constraints is searched in the search space.

In a second aspect, embodiments of the present disclosure provide an apparatus for training a prediction model, where the prediction model is used to predict the performance of a neural network structure, and the apparatus for training a prediction model includes a sampling unit configured to train through a sampling operation Prediction model; the sampling operation performed by the sampling unit includes: sampling the sub-network from the trained super-network, and training the sampled sub-network to obtain the performance information of the trained sub-network; based on the trained sub-network and the corresponding performance information to construct sample data, and use the sample data to train the prediction model; in response to determining that the accuracy of the prediction model trained in the current sampling operation does not meet the preset conditions, perform the next sampling operation, and in the next sampling operation Increase the number of sub-networks sampled.

The sampling unit for training the prediction model samples the sub-network from the trained super-network in the following manner: using the initial recurrent neural network to sample the sub-network from the trained super-network; Before the sub-network is trained, the sampling operation performed by the sampling unit further includes: generating feedback information based on the performance information of the trained sub-network, so as to iteratively update the recurrent neural network based on the feedback information; Sub-networks are sampled from the completed super-network.

The sampling unit for training the prediction model samples the sub-network from the trained super-network in the following manner: sampling the sub-network that has not been sampled from the trained super-network; and the sampling unit is constructed as follows Sample data: construct sample data based on the sub-network and corresponding performance information sampled in the current sampling operation, and the sub-network and corresponding performance information sampled in the previous sampling operation.

The sampling operation for training the prediction model further includes: in response to determining that the accuracy of the prediction model satisfies a preset condition, generating a trained prediction model based on the training result of the current sampling operation.

The apparatus for training a prediction model further includes: a search unit configured to predict the performance of the model structure in the preset model structure search space based on the trained prediction model, and the performance of the preset deep learning task scene Constraints, the neural network model structure that satisfies the performance constraints is searched in the model structure search space.

In a third aspect, embodiments of the present disclosure provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, when the one or more programs are processed by the one or more processors Execution causes the one or more processors to implement the method for training a predictive model as provided by the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, wherein the program, when executed by a processor, implements the method for training a prediction model provided in the first aspect.

The method and device for training a prediction model according to the above-mentioned embodiments of the present disclosure train the prediction model through a sampling operation; wherein the sampling operation includes: sampling a sub-network from a super-network that has been trained, and analyzing the sampled sub-network Perform training to obtain performance information of the trained sub-network; construct sample data based on the trained sub-network and the corresponding performance information, and use the sample data to train the prediction model; in response to determining that the accuracy of the prediction model does not meet the preset conditions, The next sampling operation is performed and the number of sampled sub-networks is increased in the next sampling operation, where the prediction model is used to predict the performance of the neural network structure. The method and device can obtain a prediction model for predicting the performance of any model structure, so that when applied to the automatic search of the model structure, the model structure with the best performance can be obtained by searching only once for different constraints, effectively reducing the number of The resources consumed by model structure search reduce the cost of model structure search.

Description of drawings

Other features, objects and advantages of the present disclosure will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings:

FIG. 1 is an exemplary system architecture diagram to which embodiments of the present disclosure may be applied;

2 is a flowchart of one embodiment of a method for training a predictive model according to the present disclosure;

3 is a flowchart of another embodiment of a method for training a predictive model according to the present disclosure;

4 is a schematic structural diagram of an embodiment of the apparatus for training a prediction model of the present disclosure;

FIG. 5 is a schematic structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure.

Detailed ways

The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments of the present disclosure and the features of the embodiments may be combined with each other under the condition of no conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

FIG. 1 illustrates an exemplary system architecture 100 to which a method for training a super-network or an apparatus for training a super-network of the present disclosure may be applied.

As shown in FIG. 1 , the system architecture 100 may include terminal devices 101 , 102 , and 103 , a network 104 and a server 105 . The network 104 is a medium used to provide a communication link between the terminal devices 101 , 102 , 103 and the server 105 . The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The terminal devices 101, 102, and 103 interact with the server 105 through the network 104 to receive or send messages and the like. The terminal devices 101, 102, and 103 may be client devices on which various client applications may be installed. For example, image processing applications, information analysis applications, voice assistant applications, shopping applications, financial applications, etc.

The terminal devices 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smart phones, tablet computers, e-book readers, laptop computers, desktop computers, and the like. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (eg, multiple software or software modules for providing distributed services), or as a single software or software module. There is no specific limitation here.

The server 105 may be a server running various services, such as a server running object tracking, voice processing services based on image or voice data. The server 105 can obtain deep learning task data from the terminal devices 101, 102, 103, or obtain deep learning task data from a database to construct training samples, and automatically search and optimize the model structure of the neural network used to perform the deep learning task. The server 105 may also run a prediction model for predicting the performance of the neural network structure, and predict the performance of different neural network structures based on the prediction model during automatic model structure search, so as to quickly determine the neural network model structure with the best performance.

In the application scenario of the embodiment of the present disclosure, the server 105 may realize the automatic search of the model structure of the neural network through the super network. The server 105 can train the super network based on the acquired deep learning task data, such as media data such as images, texts, and voices. After the training of the super network is completed, the server 105 can sample the sub-network structure from the super network to execute the corresponding Task.

The server 105 may also be a back-end server that provides back-end support for applications installed on the terminal devices 101 , 102 , and 103 . For example, the server 105 may receive the data to be processed sent by the terminal devices 101 , 102 , and 103 , process the data using a neural network model, and return the processing result to the terminal devices 101 , 102 , and 103 .

In an actual scenario, the terminal devices 101 , 102 , and 103 may send deep learning task requests related to tasks such as voice interaction, text classification, dialogue behavior classification, image recognition, and key point detection to the server 105 . The neural network model that has been trained for the corresponding deep learning task can be run on the server 105, and the neural network model can be used to process information.

It should be noted that the method for training a prediction model provided by the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the apparatus for training the prediction model is generally set in the server 105 .

In some scenarios, the server 105 may obtain source data (eg, training samples, trained hyper-networks, etc.) required for training the predictive model from a database, memory, or other device, in which case the exemplary system architecture 100 may not have a terminal Devices 101 , 102 , 103 and network 104 .

It should be noted that the server 105 may be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or can be implemented as a single software or software module. There is no specific limitation here.

It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

With continued reference to Figure 2, a flow 200 of one embodiment of a method for training a predictive model according to the present disclosure is shown.

The predictive models of the present disclosure are used to predict the performance of neural network structures. The performance of the neural network structure may include at least one of the following: the accuracy of the neural network structure to perform the corresponding deep learning task, the running power consumption of the neural network structure in the specified hardware or software environment, the operation of the neural network structure in the specified hardware or software environment Latency, memory usage of neural network structure in specified hardware or software environment, etc. It should be noted that, for different hardware or software environments, corresponding prediction models can be trained separately. Different prediction models can also be trained separately for different deep learning tasks.

The process 200 of the method for training a prediction model in this embodiment includes training the prediction model through sampling operations. The sampling operation includes the following steps 201 to 203 .

In step 201, a sub-network is sampled from the trained super-network, and the sampled sub-network is trained to obtain performance information of the trained sub-network.

In this embodiment, the execution body of the method for training a prediction model may acquire a pre-trained super network. The structure of the super network may be preset, including all network structures in the network structure search space, and each layer of the super network may include multiple network structure units in the network structure search space. Here, the network structural unit can be formed by a single network layer, such as a single convolutional layer, a single recurrent unit in a recurrent neural network, or a combination of multiple network layers, such as a convolutional layer, batch normalization layer, A convolutional block formed by connecting non-linear layers. In a super network, each network structural unit can be connected to all network structural units at the upper and lower layers. After the super network is trained, all internal network structures share parameters when building different sub-networks.

The sub-networks can be randomly sampled from the super-network, or the sub-networks can be sampled from the super-network using a trained recurrent neural network. It should be noted that in each sampling operation, multiple sub-networks can be sampled.

Training data can be obtained to train the sampled subnetworks. Training data can be media data such as images, text, voice, and video, or digital data such as location, price, and time, and can be determined according to the deep learning task performed. For example, if the deep learning task is an image classification task, then the training data is image data.

In this embodiment, the training data may be data with label information. During the sub-network training process, the error of the sub-network is determined based on the label information of the training data input to the sub-network, and then iteratively adjusts by means of error back propagation. The parameters of the sub-network, so that the sub-network gradually optimizes its parameters during the training process.

After the sub-network training is completed, the test data can be used to test the performance of the sub-network. The test data may also have label information, and the performance information of the trained sub-network is obtained according to the processing result of the trained sub-network on the test data and the corresponding label information.

In step 202, sample data is constructed based on the trained sub-network and corresponding performance information, and a prediction model is trained by using the sample data.

In this embodiment, paired sample data can be constructed according to the sub-network trained in step 201 and its performance information. In the sample data, the sub-network is the input information, and the corresponding performance information is the labeling information corresponding to the input information.

Optionally, a part of the sample data can be used as a training sample, and another part of the training sample can be used as a test sample.

The predictive model to be trained can be trained using sample data. The structure of the prediction model to be trained may be pre-built, for example, it may be automatically searched from the search space based on the NAS method, or it may be a network structure such as a pre-set convolutional neural network and a recurrent neural network. In this embodiment, the sub-network in the sample data can be encoded and input into the prediction model to be trained, and the prediction model to be trained can predict the performance information of the input sub-network. An objective function is constructed according to the difference between the prediction result of the prediction model to be trained on the performance information of the sub-network and the label information of the sub-network, and the parameters of the prediction model to be trained are iteratively adjusted by minimizing the objective function. When the value of the objective function converges to a preset range, or when the number of times of iteratively adjusting the parameters of the prediction model to be trained reaches a preset number of thresholds, the parameters of the prediction model to be trained can be fixed to obtain the current sampling operation. prediction model.

Since the sub-network is sampled from the trained super-network, and the initial parameters of each sub-network have high precision, the sub-network can quickly converge during training in step 202, so the solution of this embodiment can quickly complete each sub-network. sub-sampling operations, thereby speeding up the training of predictive models.

In step 203, in response to determining that the accuracy of the prediction model does not meet the preset condition, a next sampling operation is performed, and the number of sampled sub-networks is increased in the next sampling operation.

After stopping the training of the prediction model to be trained in the current sampling operation, the prediction accuracy of the prediction model trained in the current sampling operation can be tested by using the test sample. The performance information of each sub-network is predicted, and the prediction result is compared with the label information of each sub-network in the test sample to obtain the prediction accuracy of the prediction model.

If the accuracy of the prediction model trained in the current sampling operation does not meet the preset conditions, for example, does not reach the preset accuracy threshold, the next sampling operation can be performed, and the number of sampled sub-networks can be increased in the next sampling operation. The number of added sub-networks can be preset, for example, it is set that each sampling operation is increased by 500 sub-networks compared to the previous sampling operation. In this way, in two adjacent sampling operations, the number of samples of the prediction model in the latter sampling operation increases, which can achieve better accuracy than the previous sampling operation. Moreover, by gradually increasing the number of sampled sub-networks, it can be avoided that the training of the prediction model consumes too much memory resources due to the excessively large number of samples.

In the method for training a prediction model of the above-mentioned embodiment, the prediction model is trained by a sampling operation, wherein the sampling operation includes: sampling a sub-network from a super-network that has been trained, and training the sampled sub-network, and obtaining a training completed performance information of the sub-network; construct sample data based on the trained sub-network and the corresponding performance information, and use the sample data to train the prediction model; in response to determining that the accuracy of the prediction model does not meet the preset conditions, perform the next sampling operation, And increase the number of sub-networks sampled in the next sampling operation. This method can obtain a prediction model for predicting the performance of any model structure, so that when applied to the automatic search of the model structure, the model structure with the best performance can be obtained by only one search for different constraints, which effectively reduces the model structure. Searching consumes resources, reducing model structure search costs.

Optionally, the above sampling operation may further include: in response to determining that the accuracy of the prediction model meets a preset condition, generating a trained prediction model based on the training result of the current sampling operation. If the accuracy of the prediction model trained in the current sampling operation reaches the preset accuracy threshold, the prediction model can be used as the prediction model after training. In this way, after the number of sampled sub-networks and the number of samples of the prediction model are gradually increased after multiple sampling operations to gradually optimize the prediction model, the sampling operation can be stopped when the accuracy of the prediction model meets the preset conditions to avoid excessive The sampling operation consumes memory resources.

The trained prediction model can predict the performance of the preset neural network model. In actual scenarios, before the function in the application goes online, the prediction model completed by the virtual blueprint can be used to predict the performance of the neural network model used to realize the function, and the prediction result can be used as reference information to evaluate the function of the function. Stability and reliability.

In some optional implementations of the foregoing embodiments, the foregoing step of sampling a sub-network from a trained super-network includes: sampling an unsampled sub-network from the trained super-network. That is, in each sampling operation, the sub-networks that have been sampled in the previous sampling operation will not be repeatedly sampled, and each sampling operation samples a batch of new sub-networks. At this time, the above-mentioned construction of sample data based on the trained sub-network and the corresponding performance information includes: based on the sub-network and corresponding performance information sampled in the current sampling operation, and the sub-network and corresponding performance information sampled in the previous sampling operation. Performance information builds sample data. That is, the sub-network sampled by the current sampling operation can be added to the sample data constructed in the previous sampling operation to expand the sample data. In this way, computing resource consumption caused by sub-network sampling can be minimized and the number of sample data can be gradually increased, which is beneficial to reduce the memory resources occupied by the prediction model.

Optionally, in the above sampling operation, the sampling of the sub-network in step 201 may be implemented as follows: an initial recurrent neural network is used to sample the sub-network from the super-network that has been trained. Before performing step 202 to train the sampled sub-network, the sampling operation may further include: generating feedback information based on the performance information of the trained sub-network, so as to iteratively update the recurrent neural network based on the feedback information; The network resamples sub-networks from the trained super-network.

Specifically, in the process of training the prediction model, the recurrent neural network for sampling the sub-network from the super-network may also be trained. In each sampling operation, the recurrent neural network to be trained can be used to sample the sub-network, the parameters of the recurrent neural network to be trained can be randomly initialized, and then the information such as the error of the sub-network sampled by the recurrent neural network to be trained Feedback to the recurrent neural network as feedback information, so that the recurrent neural network updates parameters according to the feedback information, and resamples the sub-network.

In this way, by training the recurrent neural network based on the sub-network sampling results, the recurrent neural network can be optimized, thereby optimizing the sub-network sampling results, and further improving the prediction accuracy of the prediction model trained based on the sub-network sampling results.

Please refer to FIG. 3 , which shows a flow of another embodiment of the method for training a prediction model of the present disclosure. As shown in FIG. 3 , the process 300 of the method for training a prediction model in this embodiment includes:

Step 301, training a prediction model through sampling operation.

The sampling operation includes the following steps 3011 , 3012 and 3013 .

Step 3011: Sample sub-networks from the trained super-network, and train the sampled sub-networks to obtain performance information of the trained sub-networks.

Step 3012 , construct sample data based on the trained sub-network and corresponding performance information, and use the sample data to train a prediction model.

Step 3013 , in response to determining that the accuracy of the prediction model trained in the current sampling operation does not meet the preset condition, execute the next sampling operation, and increase the number of sampled sub-networks in the next sampling operation.

Optionally, in the above step 3011, an initial recurrent neural network can be used to sample a sub-network from the trained super-network; and before performing step 3012 to train the sampled sub-network, the sampling operation further includes: based on The performance information of the trained sub-network generates feedback information to iteratively update the recurrent neural network based on the feedback information; based on the iteratively updated recurrent neural network, the sub-network is re-sampled from the trained super-network.

Optionally, the above step 3011 of sampling a sub-network from a super-network completed by training may include: sampling an unsampled sub-network from the super-network that has been trained. And in the above step 3012, the sample data may be constructed as follows: the sample data is constructed based on the sub-network and corresponding performance information sampled in the current sampling operation, and the sub-network and corresponding performance information sampled in the previous sampling operation.

Optionally, the sampling operation may further include: in response to determining that the accuracy of the prediction model meets a preset condition, generating a trained prediction model based on the training result of the current sampling operation.

Steps 3011, 3012, and 3013 in the above sampling operation 301 are respectively consistent with steps 201, 202, and 203 in the foregoing embodiment. For the manner, reference may be made to the descriptions of the corresponding steps in the foregoing embodiments, which will not be repeated here.

In this embodiment, the method for training the prediction model further includes:

Step 302 , based on the performance prediction result of the model structure in the preset model structure search space by the trained prediction model, and the performance constraints of the preset deep learning task scenario, search the model structure search space to meet the performance constraints. Conditional neural network model structure.

The preset model structure search space can be a search space constructed for a specified deep learning task, such as a search space constructed for image processing tasks including convolutional layers, and an attention unit constructed for sequence data such as text or speech. ) search space. The prediction can be performed by using the prediction model trained in step 301 to predict the performance of each model structure in the search space. Then, the performance prediction results of each model structure are matched with the preset performance constraints of the deep learning task scenario, and the successfully matched model structure is used as the searched neural network model structure that satisfies the performance constraints. The searched neural network model structure can be used to execute the task data in the above preset deep learning task scene.

The above performance constraints may be determined by the hardware or software environment of the device running the neural network model structure. For example, if the minimum delay of running a neural network model on a chip is 0.2 seconds, a network structure that satisfies the delay condition can be searched in the search space. Alternatively, the performance constraints described above may be determined according to the requirements of the task performed by the neural network model. For example, if a function in an application needs to achieve 95% accuracy, a neural network model structure with an accuracy of not less than 95% can be searched from the above search space.

Based on the performance prediction results of the prediction model on the network structure in the search space and the preset performance constraints, a suitable neural network model structure can be quickly searched. Therefore, the prediction model can be flexibly applied to search for suitable neural network model structures in different scenarios.

Referring to FIG. 4 , as an implementation of the above method for training a prediction model, the present disclosure provides an embodiment of an apparatus for training a prediction model, which is the same as the method shown in FIGS. 2 and 3 . Corresponding to the embodiments, the apparatus can be specifically applied to various electronic devices. Among them, the prediction model is used to predict the performance of the neural network structure.

As shown in FIG. 4 , the apparatus 400 for training a prediction model in this embodiment includes a sampling unit 401 . The sampling unit 401 is configured to train a prediction model through a sampling operation; the sampling operation performed by the sampling unit includes: sampling a sub-network from the trained super-network, and training the sampled sub-network to obtain a trained sub-network based on the trained sub-network and the corresponding performance information, construct sample data, and use the sample data to train the prediction model; in response to determining that the accuracy of the prediction model trained in the current sampling operation does not meet the preset conditions, execute the following One sampling operation, and in the next sampling operation increase the number of sub-networks sampled.

In some embodiments, the above-mentioned sampling unit 401 samples the sub-networks from the trained super-network in the following manner: using an initial recurrent neural network to sample the sub-networks from the trained super-network; Before training the sub-network, the sampling operation performed by the sampling unit further includes: generating feedback information based on the performance information of the trained sub-network, so as to iteratively update the recurrent neural network based on the feedback information; Sub-networks are sampled from the trained super-network.

In some embodiments, the sampling unit 401 samples the sub-networks from the super-network that has been trained in the following manner: sampling the sub-networks that have not been sampled from the super-network that has been trained; and the sampling unit 401 according to The sample data is constructed as follows: the sample data is constructed based on the sub-network and corresponding performance information sampled in the current sampling operation, and the sub-network and corresponding performance information sampled in the previous sampling operation.

In some embodiments, the above sampling operation further includes: in response to determining that the accuracy of the prediction model meets a preset condition, generating a trained prediction model based on the training result of the current sampling operation.

In some embodiments, the above apparatus further includes: a search unit configured to predict the performance of the model structure in the preset model structure search space based on the trained prediction model, and the performance of the preset deep learning task scenario Constraints, the neural network model structure that satisfies the performance constraints is searched in the model structure search space.

The sampling unit 401 in the above apparatus 400 corresponds to the steps in the method described with reference to FIG. 2 and FIG. 3 . Therefore, the operations, features, and technical effects that can be achieved as described above with respect to the method for training a prediction model are also applicable to the apparatus 400 and the units included therein, and will not be repeated here.

Referring next to FIG. 5 , it shows a schematic structural diagram of an electronic device (eg, the server shown in FIG. 1 ) 500 suitable for implementing embodiments of the present disclosure. The electronic device shown in FIG. 5 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 5 , an electronic device 500 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 501 that may be loaded into random access according to a program stored in a read only memory (ROM) 502 or from a storage device 508 Various appropriate actions and processes are executed by the programs in the memory (RAM) 503 . In the RAM 503, various programs and data necessary for the operation of the electronic device 500 are also stored. The processing device 501 , the ROM 502 , and the RAM 503 are connected to each other through a bus 504 . An input/output (I/O) interface 505 is also connected to bus 504 .

Typically, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 507 such as a computer; a storage device 508 including, for example, a hard disk; and a communication device 509 . Communication means 509 may allow electronic device 500 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 5 shows electronic device 500 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 5 can represent one device, and can also represent multiple devices as required.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 509 , or from the storage device 508 , or from the ROM 502 . When the computer program is executed by the processing device 501, the above-described functions defined in the methods of the embodiments of the present disclosure are performed. It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Rather, in embodiments of the present disclosure, a computer-readable signal medium may include a data signal in baseband or propagated as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: trains a prediction model through a sampling operation; the sampling operation includes: from the trained super network The sub-network is sampled from the middle of the network, and the sampled sub-network is trained to obtain the performance information of the trained sub-network; the sample data is constructed based on the trained sub-network and the corresponding performance information, and the sample data is used to train the prediction model; In order to determine that the accuracy of the prediction model trained in the current sampling operation does not meet the preset conditions, perform the next sampling operation, and increase the number of sampled sub-networks in the next sampling operation, wherein the prediction model is used to predict the neural network structure. performance.

Computer program code for carrying out operations of embodiments of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, and also A conventional procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider to via Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented in software or hardware. The described unit may also be provided in the processor, for example, it may be described as: a processor includes a sampling unit. Among them, the names of these units do not constitute a limitation of the unit itself in some cases, for example, the sampling unit can also be described as "a unit for training a prediction model through sampling operations".

The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned inventive concept, the above-mentioned technical features or Other technical solutions formed by any combination of its equivalent features. For example, a technical solution is formed by replacing the above-mentioned features with the technical features disclosed in this application (but not limited to) with similar functions.

Claims (12)

1. A method for training a predictive model for predicting the performance of a neural network structure, the method comprising training the predictive model by a sampling operation;

the sampling operation comprises:

sampling a sub-network from the trained super-network, and training the sampled sub-network to obtain the performance information of the trained sub-network;

constructing sample data based on the trained sub-network and the corresponding performance information, and training the prediction model by using the sample data;

and in response to determining that the precision of the prediction model trained in the current sampling operation does not meet the preset condition, executing the next sampling operation, and increasing the number of sub-networks for sampling in the next sampling operation.

2. The method of claim 1, wherein said sampling a subnetwork from a trained super-network comprises:

sampling a sub-network from the trained super-network by adopting an initial recurrent neural network; and

before training the sampled subnetwork, the sampling operation further comprises:

generating feedback information based on the performance information of the trained sub-network to iteratively update the recurrent neural network based on the feedback information;

and re-sampling the sub-networks from the trained super-network based on the iteratively updated recurrent neural network.

3. The method of claim 1, wherein said sampling a subnetwork from a trained super-network comprises:

sampling a sub-network which is not sampled from the trained super-network; and

the constructing of sample data based on the trained sub-networks and corresponding performance information comprises:

and constructing sample data based on the sub-networks and the corresponding performance information sampled in the current sampling operation and the sub-networks and the corresponding performance information sampled in the last sampling operation.

4. The method of claim 1, wherein the sampling operation further comprises:

and generating a trained prediction model based on a training result of the current sampling operation in response to determining that the precision of the prediction model meets a preset condition.

5. The method of any of claims 1-4, wherein the method further comprises:

and searching a neural network model structure meeting the performance constraint condition in the model structure search space based on the performance prediction result of the trained prediction model on the model structure in the preset model structure search space and the performance constraint condition of the preset deep learning task scene.

6. An apparatus for training a prediction model for predicting performance of a neural network structure, the apparatus comprising a sampling unit configured to train the prediction model by a sampling operation;

the sampling operation performed by the sampling unit comprises:

sampling a sub-network from the trained super-network, and training the sampled sub-network to obtain the performance information of the trained sub-network;

constructing sample data based on the trained sub-network and the corresponding performance information, and training the prediction model by using the sample data;

and in response to determining that the precision of the prediction model trained in the current sampling operation does not meet the preset condition, executing the next sampling operation, and increasing the number of sub-networks for sampling in the next sampling operation.

7. The apparatus of claim 6, wherein the sampling unit samples a subnetwork from a trained super-network as follows:

sampling a sub-network from the trained super-network by adopting an initial recurrent neural network; and

before training the sampled sub-network, the sampling operation performed by the sampling unit further includes:

generating feedback information based on the performance information of the trained sub-network to iteratively update the recurrent neural network based on the feedback information;

and re-sampling the sub-networks from the trained super-network based on the iteratively updated recurrent neural network.

8. The apparatus of claim 6, wherein the sampling unit samples a subnetwork from a trained super-network as follows:

sampling a sub-network which is not sampled from the trained super-network; and

the sampling unit constructs sample data according to the following mode:

and constructing sample data based on the sub-networks and the corresponding performance information sampled in the current sampling operation and the sub-networks and the corresponding performance information sampled in the last sampling operation.

9. The apparatus of claim 6, wherein the sampling operation further comprises:

and generating a trained prediction model based on a training result of the current sampling operation in response to determining that the precision of the prediction model meets a preset condition.

10. The apparatus of any of claims 6-9, wherein the apparatus further comprises:

and the searching unit is configured to search out a neural network model structure meeting the performance constraint condition in the model structure searching space based on the performance prediction result of the trained prediction model on the model structure in the preset model structure searching space and the performance constraint condition of the preset deep learning task scene.

11. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.

12. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-5.

Images