我正在研究关键字提取问题。考虑非常普遍的情况

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')

t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.

"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."

"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and yet you say I am useless! Thus ungratefully, O Jupiter, do men receive their blessings!"

Our best blessings are often the least appreciated."""

tfs = tfidf.fit_transform(t.split(" "))
str = 'tree cat travellers fruit jupiter'
response = tfidf.transform([str])
feature_names = tfidf.get_feature_names()

for col in response.nonzero()[1]:
    print(feature_names[col], ' - ', response[0, col])

这给了我

  (0, 28)   0.443509712811
  (0, 27)   0.517461475101
  (0, 8)    0.517461475101
  (0, 6)    0.517461475101
tree  -  0.443509712811
travellers  -  0.517461475101
jupiter  -  0.517461475101
fruit  -  0.517461475101

这很好。对于任何新进来的文档，有没有办法获得 tfidf 分数最高的前 n 个词条？

最佳答案

您必须做一点点歌舞才能将矩阵作为 numpy 数组，但这应该可以满足您的需求:

feature_array = np.array(tfidf.get_feature_names())
tfidf_sorting = np.argsort(response.toarray()).flatten()[::-1]

n = 3
top_n = feature_array[tfidf_sorting][:n]

这给了我:

array([u'fruit', u'travellers', u'jupiter'], 
  dtype='<U13')

argsort 调用真的很有用，here are the docs for it .我们必须做 [::-1] 因为 argsort 只支持从小到大的排序。我们调用 flatten 将维度减少到 1d，以便排序索引可以用于索引 1d 特征数组。请注意，包含对 flatten 的调用仅在您同时测试一个文档时才有效。

另外，另一方面，您的意思是 tfs = tfidf.fit_transform(t.split("\n\n")) 之类的吗？否则，多行字符串中的每个术语都被视为“文档”。使用 \n\n 意味着我们实际上正在查看 4 个文档(每行一个)，当您考虑 tfidf 时，这更有意义。

关于python - Scikit Learn TfidfVectorizer : How to get top n terms with highest tf-idf score，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/34232190/