The word segmentation tool is needed when doing relationship extraction recently.

Try to segment the following sentence:

South Korea Yu had previously invited Hon Hai to invest in Kaohsiung.

After the word segmentation:

South Korea Yu had previously invited Hon Hai to invest in Kaohsiung.

The participle exists in the case of (Korea Yu) as (Korean Yu), and I hope that I do not want to separate the words "Korean Yu".

At this point you can use the dictionary function of thulac.

Sent = "Korea Yu had previously invited Hon Hai to Kaohsiung to invest."

segment = thulac.thulac(user_dict='dict.txt',seg_only=True)
thu_out = segment.cut(sent, text=True)
print(thu_out)

Among them, dict.txt is:

Korean Yu

but! ! ! but! ! ! Adding "Korean Yu" to the dictionaryNo change in results(For words in other dictionary words, the word segmentation can be successfully segmented) that the result of the word segmentation is still (Korean Yu)

I tried to expand the dictionary content, that is, South Korea Yu *3 or even *N (N is a large enough number)

The changed dictionary content is:

Korean Yu
 Korean Yu
 Korean Yu

The result is that THULAC can correctly identify "Korean Yoga"

South Korea Yu had previously invited Hon Hai to invest in Kaohsiung.

The posterior test shows that as long as two "Korean Yu" are added to the dictionary, it can be successfully identified. Since the internal model of THULAC is transparent to the user, I guess that the more the word appears in the dictionary, the more the tokenizer tends to segment the sentence by dictionary.

Similarly, I also experimented on the LTP word breaker. LTP also exists in the dictionary if "Korean Yu" only appears once, it can not be correctly segmented, and multiple times can be correctly segmented.

Test code:

import thulac
from pyltp import Segmentor
from pyltp import CustomizedSegmentor
import os

Sent = "Korea Yu had previously invited Hon Hai to Kaohsiung to invest, and on the 16th live broadcast, Han even revealed that he would meet directly with Guo Taiming on the 17th and invite him to Kaohsiung to expand investment to create more job opportunities."

segment = thulac.thulac(user_dict='dict.txt',seg_only=True)
thu_out = segment.cut(sent, text=True)
print(thu_out)

LTP_DATA_DIR = 'D:\LTP\ltp_data_v3.4.0'
cws_model_path = os.path.join(LTP_DATA_DIR, 'cws.model')
segmentor = Segmentor()
#segmentor.load_with_lexicon(cws_model_path, 'dict.txt')
segmentor.load(cws_model_path)
ltp_out = segmentor.segment(sent)
print(" ".join(w for w in ltp_out))
segmentor.release()

onthulac