我想在 Python 中设计一个自定义分词器模块，让用户可以指定用于输入的分词器。例如，考虑以下输入:

Q: What is a good way to achieve this? A: I am not so sure. I think I will use Python.

我希望能够提供 NLTK's sentence tokenization , sent_tokenize() 作为一个选项，因为它在许多情况下都能很好地工作，我不想重新发明轮子。除此之外，我还想提供一个更细粒度的标记化构建器(类似于规则引擎的东西)。让我解释一下:

假设我提供了几个分词器:

SENTENCE # Tokenizes the given input by using sent_tokenize()
WORD # Tokenizes the given input by using word_tokenize()
QA # Tokenizes using a custom regular expression. E.g., Q: (.*?) A: (.*?)

我想支持如下规则:

1. QA -> SENTENCE:首先应用 QA 分词器，然后是句子分词器
2. QA:仅应用 QA 标记器

因此，预期的输出如下:

1.质量检查 -> 句子

[
  ('QUESTION', 
             ('SENTENCE', 'What is a good way to achieve this?'), 
  ),
  ('ANSWER', 
             ('SENTENCE', 'I am not so sure', 'I think I will use Python')
  )
]

<强>2。质量检查

[
  ('QUESTION', 'What is a good way to achieve this?'),
  ('ANSWER', 'I am not so sure. I think I will use Python')
]

有什么好的设计可以有效地实现这一目标？

最佳答案

由于在 Python 中标记化很容易，我想知道您的模块计划提供什么。 我的意思是，在开始一个软件时，一个好的设计而不是首先考虑使用场景，而不是首先考虑数据结构。

您的预期输出示例有点令人困惑。 我假设您希望标记器在左侧返回名称，在右侧返回标记列表。 我玩了一点来获得类似的结果，但使用列表更容易处理:

import re

# some tokenizers
def tokzr_WORD(txt): return ('WORD', re.findall(r'(?ms)\W*(\w+)', txt))  # split words
def tokzr_SENT(txt): return ('SENTENCE', re.findall(r'(?ms)\s*(.*?(?:\.|\?|!))', txt))  # split sentences
def tokzr_QA(txt):
    l_qa = []
    for m in re.finditer(r'(?ms)^[\s#\-\*]*(?:Q|Question)\s*:\s*(?P<QUESTION>\S.*?\?)[\s#\-\*]+(?:A|Answer)\s*:\s*(?P<ANSWER>\S.*?)$', txt):  # split (Q, A) sequences
        for k in ['QUESTION', 'ANSWER']:
            l_qa.append(m.groupdict()[k])
    return ('QA', l_qa)
def tokzr_QA_non_canonical(txt):  # Note: not supported by tokenize_recursively() as not canonical.
    l_qa = []
    for m in re.finditer(r'(?ms)^[\s#\-\*]*(?:Q|Question)\s*:\s*(?P<QUESTION>\S.*?\?)[\s#\-\*]+(?:A|Answer)\s*:\s*(?P<ANSWER>\S.*?)$', txt):  # split (Q, A) sequences
        for k in ['QUESTION', 'ANSWER']:
            l_qa.append((k, m.groupdict()[k]))
    return l_qa

dict_tokzr = {  # control string: tokenizer function
    'WORD'    : tokzr_WORD,
    'SENTENCE': tokzr_SENT,
    'QA'      : tokzr_QA,
}

# the core function
def tokenize_recursively(l_tokzr, work_on, lev=0):
    if isinstance(work_on, basestring):
        ctrl, work_on = dict_tokzr[l_tokzr[0]](work_on)  # tokenize
    else:
        ctrl, work_on = work_on[0], work_on[1:]  # get right part
    ret = [ctrl]
    if len(l_tokzr) == 1:
        ret.append(work_on)  # add right part
    else:
        for wo in work_on:  # dive into tree
            t = tokenize_recursively(l_tokzr[1:], wo, lev + 1)
            ret.append(t)
    return ret

# just for printing
def nestedListLines(aList, ind='    ', d=0):
    """ Returns multi-line string representation of \param aList.  Use \param ind to indent per level. """
    sRet = '\n' + d * ind + '['
    nested = 0
    for i, e in enumerate(aList):
        if i:
            sRet += ', '
        if type(e) == type(aList):
            sRet += nestedListLines(e, ind, d + 1)
            nested = 1
        else:
            sRet += '\n' + (d + 1) * ind + repr(e) if nested else repr(e)
    sRet += '\n' + d * ind + ']' if nested else ']'
    return sRet

# main()
inp1 = """
    * Question: I want try something.  Should I?
    * Answer  : I'd assume so.  Give it a try.
"""
inp2 = inp1 + 'Q: What is a good way to achieve this?  A: I am not so sure. I think I will use Python.'
print repr(tokzr_WORD(inp1))
print repr(tokzr_SENT(inp1))
print repr(tokzr_QA(inp1))
print repr(tokzr_QA_non_canonical(inp1))  # Really this way?
print

for ctrl, inp in [  # example control sequences
    ('SENTENCE-WORD', inp1),
    ('QA-SENTENCE', inp2)
]:
    res = tokenize_recursively(ctrl.split('-'), inp)
    print nestedListLines(res)

顺便说一句。 Python/Lib/tokenize.py(用于 Python 代码本身)可能值得一看如何处理。

关于python - 用 Python 编写分词器，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/15929233/