CN109388806B - Chinese word segmentation method based on deep learning and forgetting algorithm - Google Patents

Abstract

The invention discloses a Chinese word segmentation method based on deep learning and forgetting algorithm, which includes the following steps: 1: Scanning sentences word by word to obtain natural language, and using deep learning word segmentation method to divide the scanned natural language into word sequences and income to the first word second: scan the sentence word by word to obtain natural language, and use the forgetting algorithm word segmentation method to hyphenate the obtained natural language into candidate words and store them in the second thesaurus; third: combine the word sequence in the first thesaurus with the second The candidate words in the thesaurus are fused to obtain the final word segmentation results. The fusion method is: if the continuous words in the second thesaurus correspond to words in deep learning, they will be merged into words; if a single word in the second thesaurus corresponds to deep learning If it is a word in learning, it will be merged forward or backward into a word. The word segmentation method of the present invention can automatically detect domain knowledge by combining the deep learning word segmentation method and the forgetting algorithm word segmentation method, complete the new word discovery function in the unsupervised field, and improve the word segmentation effect.

Description

A Chinese word segmentation method based on deep learning and forgetting algorithm

technical field

The invention relates to the technical field of word segmentation, in particular to a Chinese word segmentation method based on deep learning and forgetting algorithms.

Background technique

Chinese Word Segmentation refers to dividing a sequence of Chinese characters into individual words. Word segmentation is the process of recombining continuous word sequences into word sequences according to certain specifications.

1. Word segmentation method based on string matching

This method is also called the mechanical word segmentation method. It matches the Chinese character string to be analyzed with an entry in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the match is successful. (a word is recognized). According to different scanning directions, the string matching word segmentation method can be divided into forward matching and reverse matching; according to the priority matching of different lengths, it can be divided into maximum (longest) matching and minimum (shortest) matching; according to whether it is compatible with the part-of-speech tagging process It can be divided into a simple word segmentation method and an integrated method combining word segmentation and labeling. Several commonly used mechanical word segmentation methods are as follows:

1) forward maximum matching method (direction from left to right);

2) reverse maximum matching method (direction from right to left);

3) Minimal segmentation (making the number of words cut out in each sentence minimum).

The above various methods can also be combined with each other, for example, the forward maximum matching method and the reverse maximum matching method can be combined to form a two-way matching method. Due to the characteristics of Chinese characters into words, forward minimum matching and reverse minimum matching are generally seldom used. Generally speaking, the segmentation accuracy of reverse matching is slightly higher than that of forward matching, and there are fewer ambiguities encountered. The statistical results show that the error rate of purely using forward maximum matching is 1/169, and the error rate of purely using reverse maximum matching is 1/245. But this precision is far from meeting the actual needs. The word segmentation systems actually used all use mechanical word segmentation as a means of initial segmentation, and it is necessary to use various other language information to further improve the accuracy of segmentation.

One method is to improve the scanning method, which is called feature scanning or flag segmentation, and firstly identify and segment some words with obvious characteristics in the string to be analyzed. Using these words as breakpoints, the original string can be divided into Then perform mechanical word segmentation for smaller strings, thereby reducing the error rate of matching. Another method is to combine word segmentation and part-of-speech tagging, use rich part-of-speech information to help word segmentation decision-making, and in turn check and adjust the word segmentation results during the tagging process, thereby greatly improving the accuracy of segmentation.

For the mechanical word segmentation method, a general model can be established. There are professional academic papers in this area, which will not be discussed in detail here.

2. Comprehension-based word segmentation method

This word segmentation method achieves the effect of recognizing words by letting the computer simulate the human understanding of the sentence. Its basic idea is to perform syntactic and semantic analysis at the same time of word segmentation, and use syntactic information and semantic information to deal with ambiguity. It usually includes three parts: the word segmentation subsystem, the syntax and semantics subsystem, and the general control part. Under the coordination of the general control part, the word segmentation subsystem can obtain syntactic and semantic information about words and sentences to judge the ambiguity of word segmentation, that is, it simulates the process of human understanding of sentences. This word segmentation method requires the use of a large amount of language knowledge and information. Due to the generality and complexity of Chinese language knowledge, it is difficult to organize various language information into a form that can be directly read by machines. Therefore, the word segmentation system based on comprehension is still in the experimental stage.

3. Statistical word segmentation method

In terms of form, a word is a combination of stable characters, so in the context, the more times adjacent characters appear at the same time, the more likely it is to form a word. Therefore, the frequency or probability of adjacent co-occurrence of words can better reflect the credibility of words. The frequency of combinations of adjacent co-occurring characters in the corpus can be counted, and their mutual occurrence information can be calculated. Define the mutual occurrence information of two characters, and calculate the adjacent co-occurrence probability of two Chinese characters X and Y. Mutual appearance information reflects the closeness of the combination relationship between Chinese characters. When the degree of closeness is higher than a certain threshold, it can be considered that this word group may form a word. This method only needs to count the frequency of words in the corpus, and does not need to segment the dictionary, so it is also called the dictionary-free word segmentation method or the statistical word extraction method. However, this method also has certain limitations. It will often extract some common word groups that have a high co-occurrence frequency but are not words, such as "one", "one", "some", "my", "Many", etc., and the recognition accuracy of common words is poor, and the time and space overhead is large. The statistical word segmentation system used in practice must use a basic word segmentation dictionary (common word dictionary) for string matching word segmentation, and at the same time use statistical methods to identify some new words, that is, to combine string frequency statistics and string matching. It has the characteristics of fast speed and high efficiency, and also utilizes the advantages of no-dictionary word segmentation combined with contextual recognition of new words and automatic disambiguation.

1. Ambiguity recognition

Ambiguity refers to the same sentence, there may be two or more segmentation methods. For example: surface, because "surface" and "surface" are both words, then this phrase can be divided into "surface" and "surface". This is called intersection ambiguity. This kind of cross-ambiguity is very common. The example of "kimono" mentioned above is actually an error caused by cross-ambiguity. "Makeup and clothing" can be divided into "makeup and clothing" or "makeup and clothing". Since there is no human knowledge to understand, it is difficult for the computer to know which solution is correct.

Crossover ambiguity is easier to deal with than combination ambiguity, and combination ambiguity must be judged based on the entire sentence. For example, in the sentence "This doorknob is broken," "handle" is a word, but in the sentence "Please remove your hand," "handle" is not a word; in the sentence "The general appointed a lieutenant general" , "Lieutenant General" is a word, but in the sentence "production will triple in three years", "Lieutenant General" is no longer a word. How do computers recognize these words?

If both cross-ambiguity and combined ambiguity can be solved by computers, there is still a difficult problem in ambiguity, which is true ambiguity. True ambiguity means that given a sentence, it is up to people to judge which should be a word and which should not be a word. For example: "The table tennis auction is over", which can be divided into "The table tennis auction is over", and it can also be divided into "The table tennis auction is over". one word.

2. New word recognition

New words, technical terms are called unregistered words. That is, those words that have not been included in the dictionary, but can indeed be called words. The most typical is the name of a person, which can be easily understood by humans. In the sentence "Wang Junhu has gone to Guangzhou", "Wang Junhu" is a word because it is a person's name, but it is difficult for a computer to recognize it. If "Wang Junhu" is included in the dictionary as a word, there are so many names in the world, and there are new names added every moment. Including these names is a huge project in itself. Even if this work can be done, there will still be problems, for example: in the sentence "Wang Junhu has a tiger head and a tiger brain", can "Wang Junhu" still count as a word?

In addition to personal names, new words include institution names, place names, product names, brand names, abbreviations, abbreviations, etc., which are difficult to deal with, and these happen to be words that people often use, so for search engines , new word recognition in word segmentation system is very important. At present, the accuracy of new word recognition has become one of the important symbols to evaluate the quality of a word segmentation system. The existing word segmentation algorithm is based on thesaurus, and words that do not appear in the thesaurus cannot be segmented.

Contents of the invention

Aiming at the above technical problems, the present invention provides a Chinese word segmentation method based on deep learning and forgetting algorithm. By combining the deep learning word segmentation method and the forgetting algorithm word segmentation method, it can automatically detect domain knowledge and complete the new word discovery function in the unsupervised field. , improve word segmentation effect.

In order to solve the above technical problems, the technical solution adopted in the present invention is: a Chinese word segmentation method based on deep learning and forgetting algorithm, comprising the following steps:

Step 1: Scan the sentence word by word to obtain natural language, and use the deep learning word segmentation method to divide the scanned natural language into word sequences and store them in the first thesaurus;

Step 2: Scan the sentence word by word to obtain natural language, use the forgetting algorithm word segmentation method to hyphenate the obtained natural language into candidate words and enter them into the second thesaurus,

Step 3: Fuse the word sequence in the first thesaurus with the candidate words in the second thesaurus to obtain the final word segmentation result, wherein the fusion method is:

Both the first thesaurus and the second thesaurus are words, they will be merged into words; both the first thesaurus and the second thesaurus are single characters, then they will be merged into words; the continuous words in the second thesaurus, if they correspond to the deep learning If it is a word in the second lexicon, it will be merged into a word; if a single word in the second lexicon corresponds to a word in deep learning, it will be merged into a word forward or backward.

Wherein, the step 1 deep learning word segmentation method adopts the RNN method.

Wherein, the step 1 deep learning word segmentation method adopts the LSTM model in the RNN method.

Wherein, the step 2 forgetting algorithm word segmentation method adopts the judgment formula as follows:

P(W _n W _n+1 )＜P(W _n )*P(W _n+1 )

Among them, Wn is the nth word in the scanned sentence

Wherein, the forgetting curve adopted by the forgetting algorithm in step 2 is the Newton cooling curve.

The beneficial effects of the present invention are:

The word segmentation method of the present invention has the following advantages:

(1) Unsupervised learning, which can use a large amount of corpus for training;

(2) O(N)-level time complexity, which can be completed in a relatively short time for large-scale word segmentation;

(3) Thesaurus self-maintenance, the program can discover and add new words, adjust word frequency, clean up wrong words, remove uncommon words, and keep the dictionary size appropriate without manual participation;

(4) Domain self-adaptation: when the domain changes, entries and word frequency are adaptively adjusted accordingly;

(5) It can support the word segmentation of the proprietary thesaurus such as names of uncommon artists and program titles.

Description of drawings

Fig. 1 is the forgetting curve figure that forgetting coefficient adopts in a kind of Chinese word segmentation method based on deep learning and forgetting algorithm of the present invention;

Fig. 2 is a logic diagram of the LSTM model in a Chinese word segmentation method based on deep learning and forgetting algorithm of the present invention.

Detailed ways

In the following detailed description of the preferred embodiment, reference is made to the accompanying drawings which form a part hereof. The accompanying drawings show, by way of example, specific embodiments in which the invention can be practiced. The illustrated embodiments are not intended to be exhaustive of all embodiments in accordance with the invention. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. Accordingly, the following detailed description is not limiting, and the scope of the invention is defined by the appended claims.

A Chinese word segmentation method based on deep learning and forgetting algorithm, comprising the following steps:

Step 1: Scan the sentence word by word to obtain natural language, and use the deep learning word segmentation method to divide the scanned natural language into word sequences and store them in the first thesaurus;

Step 2: Scan the sentence word by word to obtain natural language, use the forgetting algorithm word segmentation method to hyphenate the obtained natural language into candidate words and enter them into the second thesaurus,

Step 3: Fuse the word sequence in the first thesaurus with the candidate words in the second thesaurus to obtain the final word segmentation result, wherein the fusion method is:

Both the first thesaurus and the second thesaurus are words, they will be merged into words; both the first thesaurus and the second thesaurus are single characters, then they will be merged into words; the continuous words in the second thesaurus, if they correspond to the deep learning If it is a word in the second lexicon, it will be merged into a word; if a single word in the second lexicon corresponds to a word in deep learning, it will be merged into a word forward or backward.

The invention adopts the combination of deep learning and forgetting algorithm, which can automatically detect the domain knowledge, complete the function of discovering new words in the unsupervised domain, and improve the effect of word segmentation.

The main steps of the forgetting algorithm are as follows:

Use the following steps to complete word segmentation in O(N)-level time with a single pass scan:

Scan the sentence word by word, find all the words ending with the word within the limited word length from the thesaurus, calculate the product of the probability of the word in it and the words before the word, take the word with the largest result value, and cache the current word respectively The maximum probability product of the location, and the corresponding word segmentation results. Repeat the above steps until the sentence is scanned, and the result of the word segmentation of the last word is the word segmentation result of the whole sentence.

If the adjacent two words are irrelevant, the middle of the two words can be disconnected. Scan the sentence word by word, and if the two adjacent words satisfy the following formula, then the two words will be disconnected. In this way, the sentence can be cut into several substrings to obtain the "candidate word" set. The judgment formula is shown in the figure below:

P(W _n W _n+1 )＜P(W _n )*P(W _n+1 )

Among them, Wn is the nth word in the scanned sentence

The parameters required in the formula can be obtained through statistics: by traversing the corpus once, the "frequency of a word", "frequency of co-occurrence of two adjacent words", and "sum of frequencies of all words" required in the formula can be obtained.

The forgetting coefficient used in the forgetting curve is shown in Figure 1:

The deep learning method uses the RNN method, specifically the LSTM model.

Chinese word segmentation is to divide the natural language text into a sequence of words, preferably sequence labeling, that is, use the four labels of BMES to label each word in the sentence (B is the beginning of the word, M is the middle of the word, E is the end of the word, S is the word word).

For {JD Search and Big Data Platform Data Mining Algorithm Department}

It is labeled {BE BE S BME BE BMME BME}

The original input sequence and output sequence are used to mark the corpus for training, and finally the word segmentation sequence is generated. The logic diagram of the LSTM model is shown in Figure 2. In Figure 2, X is the input sequence, and H is the output sequence. The basic idea of word segmentation is to use the sequence labeling problem to mark each word in a sentence into four BEMS labels. The input of the model is a sequence of characters, and the output is a sequence of labels, so this is a standard sequence to sequence problem.

The combined word segmentation method improves the effect of word segmentation by fusing the results of the two methods, and takes the forgetting algorithm as the theme, because

Variety shows, artist names, etc. play an important role in the recommendation

The forgetting algorithm is used as unsupervised learning, and the cost of obtaining training corpus is low

·Deep learning algorithm training corpus is scarce and the training time is long

Merger plan

· Forget the continuous words in the algorithm results, if they correspond to words in deep learning, they will be merged into words

·Forgetting a single word in the algorithm result, if it corresponds to a word in deep learning, it will be merged forward or backward into a word

·Merge with reference to part of speech

Example 1:

Obtain natural language by scanning sentences, and then use forgetting algorithm and deep learning to segment words, and the fused word segmentation results:

The following are the results of the word segmentation of the two algorithms:

The forgetting algorithm improves word segmentation results;

<Real shot> <man> <subway> <obscene> <female> <passenger> <> <being> <enthusiastic> <passenger> <twisting> <obtaining>

<Pokemon> <Network><Edition><Registration><Download><Tutorial><Video>

<Huo> <some> <some> <snowy night> <seeking> <true love> <super> <romantic> <confession> <girl> <moved> <cry> <161105><very><perfect>

<Small smile is alluring> <Zheng Mou> <Yang Mou> <kiss scene> <play> <game> <talk> <love>

<Jiang Moumou><ridicule><mahjong><should><enter><Olympic><joke><may><and><gymnastics><combination>

<week> <some> <some> <keep> <preparation> <stand> <left> <outside> <field> <ball> <field> <horn> <sound> <super big> <self> <preparation> <coup ><defence><enemy><army>;

Word segmentation results of deep learning algorithm:

<actual> <shoot> <man> <subway> <obscene woman> <passenger> <been> <enthusiastic> <passenger> <wrestle>

<Pocket> <Yokai> <Online Version> <The> <Register> <Download> <Tutorial> <Video>

<Huo Moumou><Snow><Night Seeking><True Love><Super><Romantic><Confession><Girl><Moving Pain><Crying><161105><Very><Perfect>

<微微><一笑><very><Qingcheng><Zheng Mou><Yang Mou><kiss><play><play><game><love in love>

<Jiang> <some> <some> <ridicule> <mahjong> <should> <enter the Olympics> <laughing> <may> <and> <gymnastics><combination>

<Zhou Moumou> <Defense> <Stand> <Left> <Outside> <Field Field> <Horn> <Sound super loud> <Self-prepared> <Wonderful> <Recruitment> <Enemy>;

The result after combining the above schemes:

<Real shot> <man> <subway> <obscene woman> <passenger> <been> <enthusiastic> <passenger> <wrestled>

<Pokemon> <online version> <of> <registration> <download> <tutorial> <video>

<Huo Moumou><Snow Night><Looking><True Love><Super><Romantic><Confession><Girl><Moving><Crying><161105><Very><Perfect>

<Small smile is very charming> <Zheng Mou> <Yang Mou> <kiss scene> <play> <game> <fall in love>

<Jiang Moumou><ridicule><mahjong><should><enter><Olympic><joke><may><and><gymnastics><combination>

<Zhou Moumou> <defense> <stand> <left> <outside> <field field> <horn> <sound super loud> <self-preparation> <coup> <defense> <enemy>.

Claims (3)

1. A Chinese word segmentation method based on deep learning and forgetting algorithm is characterized by comprising the following steps:

step one: the method comprises the steps of scanning sentences word by word to obtain natural language, dividing the scanned natural language into word sequences by adopting a deep learning word segmentation method, and collecting the word sequences into a first word stock;

step two: the sentence is scanned word by word to obtain natural language, the word segmentation method of forgetting algorithm is adopted to divide the obtained natural language into candidate words and receive the candidate words into a second word stock,

step three: fusing the word sequence in the first word bank with the candidate words in the second word bank to obtain a final word segmentation result, wherein the fusion method comprises the following steps:

the first word stock and the second word stock are words, and then are combined into words; the first word stock and the second word stock are single words, and are combined into words; the continuous single words in the second word stock are combined into words if the words in the corresponding deep learning; if the single word in the second word stock is the word in the corresponding deep learning, merging the single word forwards or backwards into the word;

the word segmentation method of the forgetting algorithm adopts a judgment formula as follows:

P(W _n W _n+1 )＜P(W _n )*P(W _n+1 )

wherein Wn is the nth word in the scanned sentence

P(W _n )：

P(W _n Wn ₊₁ )：

And in the second step, the forgetting curve adopted by the forgetting algorithm is a Newton cooling curve.

2. The method for Chinese word segmentation based on deep learning and forgetting algorithm as set forth in claim 1, wherein the step one deep learning word segmentation method adopts RNN method.

3. The method for Chinese word segmentation based on deep learning and forgetting algorithm according to claim 1 or 2, wherein the step one deep learning word segmentation method adopts an LSTM model in the RNN method.

Images