spacy库的使用

1.下载模型

下载对应的github地址为https://blog.51cto.com/transfer?https://github.com/explosion/spacy-models/tags

2.加载模型

eng = spacy.load("en_core_web_md-3.7.1")  # Load the English model to tokenize English text
zh = spacy.load("zh_core_web_md-3.7.0")

3.使用模型

def engTokenize(text):
    """
    Tokenize an English text and return a list of tokens
    """
    engTokenList = [token.text for token in eng.tokenizer(text)]
    return engTokenList


def zhTokenize(text):
    """
    Tokenize a German text and return a list of tokens
    """
    return [token.text for token in zh.tokenizer(text)]


print(engTokenize("Have a good day!!!"))
print(zhTokenize("我今天过的很开心，谭国军去哪玩 !!!"))