基于关联规则算法实现电影推荐系统

• 利用数据挖掘算法中的Apriori(关联规则)算法来实现一个电影推荐系统
  • 加载数据
  • 数据预处理
  • 生成频繁项集、关联规则
• 通过关联规则生成电影推荐的列表

Apriori算法

• 案例：
  啤酒与尿布: 沃尔玛超市在分析销售记录时，发现了啤酒与尿布经常一起被购买，于是他们调整了货架将两者放在了一起，结果真的提升了啤酒的销量。 原因解释: 爸爸在给宝宝买尿布的时候，会顺便给自己买点啤酒？

• 概述：
  Apriori算法是一种最有影响力的挖掘布尔关联规则的频繁项集的算法，其命名Apriori源于算法使用了频繁项集性质的先验(Prior)知识。
  接下来我们将以超市订单的例子理解关联分析相关的重要概念: Support(支持度)、Confidence(置信度)、Lift(提升度）。

  • Support(支持度)：指某事件出现的概率，在本例中即指某个商品组合出现的次数占总次数的比例。

  例：Support('Bread') = 4/5 = 0.8 Support('Milk') = 4/5 = 0.8
  Support('Bread+Milk') = 3/5 = 0.6

  • Confidence(置信度)：本质上是个条件概率，即当购买了商品A的前提下，购买商品B的概率。

  例：Confidence('Bread'—> 'Milk') = Support('Bread+Milk')/ Support('Bread') = 0.6/0.8 = 0.75

  • Lift(提升度）: 指商品A的出现，对商品B的出现的概率的提升程度。Lift(A->B) = Confidence(A, B) / Support(B)

  例：Lift('Bread'—> 'Milk') = 0.75/0.8 = 0.9375

• 对于Lift(提升度）有三种情况：

  • Lift(A->B)>1: 代表A对B的出现概率有提升。
  • Lift(A->B)=1: 代表A对B的出现概率没有提升，也没有下降。
  • Lift(A->B)<1: 代表A对B的出现概率有下降效果。

• 原理：
  该算法挖掘关联规则的过程，即是查找频繁项集(frequent itemset)的过程:

  • 频繁项集：支持度大于等于最小支持度(Min Support)阈值的项集。
  • 非频繁集：支持度小于最小支持度的项集。

• 流程：
  K = 1, 计算K项集的支持度；
  筛选掉小于最小支持度的项集；
  如果项集为空，则对应K-1项集的结果为最终结果。否则K = K+1重复2-3步

import pandas as pd
import matplotlib.pyplot as plt
import mlxtend
import numpy as np

电影数据准备

movie_data_file = './movie_dataset/movies_metadata.csv'
ratings_file = './movie_dataset/ratings_small.csv'

movie_data_df = pd.read_csv(movie_data_file)
ratings_df = pd.read_csv(ratings_file)

c:\users\ysilhouette\documents\pyenv\py3.6.5\lib\site-packages\IPython\core\interactiveshell.py:3072: DtypeWarning: Columns (10) have mixed types.Specify dtype option on import or set low_memory=False.
  interactivity=interactivity, compiler=compiler, result=result)

movie_data_df.head(5)

movie_data_df.describe()

movie_data_df.info

<bound method DataFrame.info of        adult                              belongs_to_collection    budget  \
0      False  {'id': 10194, 'name': 'Toy Story Collection', ...  30000000   
1      False                                                NaN  65000000   
2      False  {'id': 119050, 'name': 'Grumpy Old Men Collect...         0   
3      False                                                NaN  16000000   
4      False  {'id': 96871, 'name': 'Father of the Bride Col...         0   
...      ...                                                ...       ...   
45461  False                                                NaN         0   
45462  False                                                NaN         0   
45463  False                                                NaN         0   
45464  False                                                NaN         0   
45465  False                                                NaN         0   

                                                  genres  \
0      [{'id': 16, 'name': 'Animation'}, {'id': 35, '...   
1      [{'id': 12, 'name': 'Adventure'}, {'id': 14, '...   
2      [{'id': 10749, 'name': 'Romance'}, {'id': 35, ...   
3      [{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...   
4                         [{'id': 35, 'name': 'Comedy'}]   
...                                                  ...   
45461  [{'id': 18, 'name': 'Drama'}, {'id': 10751, 'n...   
45462                      [{'id': 18, 'name': 'Drama'}]   
45463  [{'id': 28, 'name': 'Action'}, {'id': 18, 'nam...   
45464                                                 []   
45465                                                 []   

                                   homepage      id    imdb_id  \
0      http://toystory.disney.com/toy-story     862  tt0114709   
1                                       NaN    8844  tt0113497   
2                                       NaN   15602  tt0113228   
3                                       NaN   31357  tt0114885   
4                                       NaN   11862  tt0113041   
...                                     ...     ...        ...   
45461  http://www.imdb.com/title/tt6209470/  439050  tt6209470   
45462                                   NaN  111109  tt2028550   
45463                                   NaN   67758  tt0303758   
45464                                   NaN  227506  tt0008536   
45465                                   NaN  461257  tt6980792   

      original_language               original_title  \
0                    en                    Toy Story   
1                    en                      Jumanji   
2                    en             Grumpier Old Men   
3                    en            Waiting to Exhale   
4                    en  Father of the Bride Part II   
...                 ...                          ...   
45461                fa                      رگ خواب   
45462                tl          Siglo ng Pagluluwal   
45463                en                     Betrayal   
45464                en          Satana likuyushchiy   
45465                en                     Queerama   

                                                overview  ... release_date  \
0      Led by Woody, Andy's toys live happily in his ...  ...   1995-10-30   
1      When siblings Judy and Peter discover an encha...  ...   1995-12-15   
2      A family wedding reignites the ancient feud be...  ...   1995-12-22   
3      Cheated on, mistreated and stepped on, the wom...  ...   1995-12-22   
4      Just when George Banks has recovered from his ...  ...   1995-02-10   
...                                                  ...  ...          ...   
45461        Rising and falling between a man and woman.  ...          NaN   
45462  An artist struggles to finish his work while a...  ...   2011-11-17   
45463  When one of her hits goes wrong, a professiona...  ...   2003-08-01   
45464  In a small town live two brothers, one a minis...  ...   1917-10-21   
45465  50 years after decriminalisation of homosexual...  ...   2017-06-09   

           revenue runtime                                   spoken_languages  \
0      373554033.0    81.0           [{'iso_639_1': 'en', 'name': 'English'}]   
1      262797249.0   104.0  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...   
2              0.0   101.0           [{'iso_639_1': 'en', 'name': 'English'}]   
3       81452156.0   127.0           [{'iso_639_1': 'en', 'name': 'English'}]   
4       76578911.0   106.0           [{'iso_639_1': 'en', 'name': 'English'}]   
...            ...     ...                                                ...   
45461          0.0    90.0             [{'iso_639_1': 'fa', 'name': 'فارسی'}]   
45462          0.0   360.0                  [{'iso_639_1': 'tl', 'name': ''}]   
45463          0.0    90.0           [{'iso_639_1': 'en', 'name': 'English'}]   
45464          0.0    87.0                                                 []   
45465          0.0    75.0           [{'iso_639_1': 'en', 'name': 'English'}]   

         status                                            tagline  \
0      Released                                                NaN   
1      Released          Roll the dice and unleash the excitement!   
2      Released  Still Yelling. Still Fighting. Still Ready for...   
3      Released  Friends are the people who let you be yourself...   
4      Released  Just When His World Is Back To Normal... He's ...   
...         ...                                                ...   
45461  Released         Rising and falling between a man and woman   
45462  Released                                                NaN   
45463  Released                             A deadly game of wits.   
45464  Released                                                NaN   
45465  Released                                                NaN   

                             title  video vote_average vote_count  
0                        Toy Story  False          7.7     5415.0  
1                          Jumanji  False          6.9     2413.0  
2                 Grumpier Old Men  False          6.5       92.0  
3                Waiting to Exhale  False          6.1       34.0  
4      Father of the Bride Part II  False          5.7      173.0  
...                            ...    ...          ...        ...  
45461                       Subdue  False          4.0        1.0  
45462          Century of Birthing  False          9.0        3.0  
45463                     Betrayal  False          3.8        6.0  
45464             Satan Triumphant  False          0.0        0.0  
45465                     Queerama  False          0.0        0.0  

[45466 rows x 24 columns]>

movie_data_df.count()

adult                    45466
belongs_to_collection     4494
budget                   45466
genres                   45466
homepage                  7782
id                       45466
imdb_id                  45449
original_language        45455
original_title           45466
overview                 44512
popularity               45461
poster_path              45080
production_companies     45463
production_countries     45463
release_date             45379
revenue                  45460
runtime                  45203
spoken_languages         45460
status                   45379
tagline                  20412
title                    45460
video                    45460
vote_average             45460
vote_count               45460
dtype: int64

movie_data_df.columns

Index(['adult', 'belongs_to_collection', 'budget', 'genres', 'homepage', 'id',
       'imdb_id', 'original_language', 'original_title', 'overview',
       'popularity', 'poster_path', 'production_companies',
       'production_countries', 'release_date', 'revenue', 'runtime',
       'spoken_languages', 'status', 'tagline', 'title', 'video',
       'vote_average', 'vote_count'],
      dtype='object')

ratings_df.head(5)

ratings_df.columns

Index(['userId', 'movieId', 'rating', 'timestamp'], dtype='object')

ratings_df.count()

userId       100004
movieId      100004
rating       100004
timestamp    100004
dtype: int64

ratings_df.shape

(100004, 4)

movie_data_df.shape

(45466, 24)

数据预处理

• 缺失值处理
• 数据去重
• 电影源信息 merge 电影评分信息

movie_data_df_t=movie_data_df[['title','id']]

movie_data_df_t.dtypes

title    object
id       object
dtype: object

ratings_df_s = ratings_df.drop(['timestamp'], axis=1)  #axis=0 跨列删除行 ，axis=1 跨行删除列

ratings_df_s.dtypes

userId       int64
movieId      int64
rating     float64
dtype: object

缺失值处理

• pandas中用NaN(Not a Number)表示浮点数和非浮点数数组中的缺失值，同时python中None值也被当作缺失值。

# pd.to_numeric 将id列 的数据 由字符串转为数值类型， 不能转换的数据设置为NaN
pd.to_numeric(movie_data_df_t['id'],errors='coerce')

0           862.0
1          8844.0
2         15602.0
3         31357.0
4         11862.0
           ...   
45461    439050.0
45462    111109.0
45463     67758.0
45464    227506.0
45465    461257.0
Name: id, Length: 45466, dtype: float64

#np.where返回满足（）内条件的数据所在的位置
np.where(pd.to_numeric(movie_data_df_t['id'], errors='coerce').isna()) #返回缺失值的位置，其中isna() 对于NaN返回True，否则返回False

(array([19730, 29503, 35587], dtype=int64),)

• loc works on labels in the index.
  • loc为Selection by Label函数，即为按标签取数据，标签是什么，就是上面的’0’~‘4’, ‘A’~‘B’。
  • 例如第一个参数选择index，第二个参数选择column
  • 建议写df.loc[0, :]，这样可以清楚的看出为第0行的所有记录，同样如果取第’A’列的所有记录，可以写df.loc[:, ‘A’]，如下图：
  • :表示所有，[]里边为先行后列
• iloc works on the positions in the index (so it only takes integers).
  • iloc函数为Selection by Position，即按位置选择数据，即第n行，第n列数据，只接受整型参数, 比如 0:2为左闭右开区间，即取0，1
  • 若要取第一列的所有数据，则为df.iloc[:, 0]，不接受’A’作为参数

movie_data_df_t.iloc[19730]

title           NaN
id       1997-08-20
Name: 19730, dtype: object

movie_data_df_t.iloc[[19730,29503,35587]]

# 将格式转换后的数据 赋值给id列
movie_data_df_t['id'] = pd.to_numeric(movie_data_df_t['id'], errors='coerce')

c:\users\ysilhouette\documents\pyenv\py3.6.5\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

​

movie_data_df_t.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 45466 entries, 0 to 45465
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   title   45460 non-null  object 
 1   id      45463 non-null  float64
dtypes: float64(1), object(1)
memory usage: 710.5+ KB

movie_data_df_t.iloc[[19730,29503,35587]]

movie_data_df_t.shape

(45466, 2)

movie_data_df_t.drop(np.where(movie_data_df_t['id'].isna())[0], inplace=True)

c:\users\ysilhouette\documents\pyenv\py3.6.5\lib\site-packages\pandas\core\frame.py:4174: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,

movie_data_df_t.shape

(45463, 2)

数据去重

movie_data_df_t.duplicated(['id','title']).sum()

30

movie_data_df_t.drop_duplicates(['id'],inplace=True)

c:\users\ysilhouette\documents\pyenv\py3.6.5\lib\site-packages\ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

movie_data_df_t.shape

(45433, 2)

ratings_df_s.duplicated(['userId','movieId']).sum()

0

movie_data_df_t['id'] = movie_data_df_t['id'].astype(np.int64)

c:\users\ysilhouette\documents\pyenv\py3.6.5\lib\site-packages\ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

movie_data_df_t.dtypes

title    object
id        int64
dtype: object

ratings_df_s.dtypes

userId       int64
movieId      int64
rating     float64
dtype: object

数据合并

# 左dataframe 和 右dataframe 根据 movieId 和 id进行合并
ratings_df_s = pd.merge(ratings_df_s,movie_data_df_t, left_on='movieId',right_on='id')

ratings_df_s.head()

ratings_df_s.drop(['id'],axis=1,inplace=True)

ratings_df_s

ratings_df_s.shape

(44989, 4)

# 有评分记录的电影的个数
len(ratings_df_s['title'].unique())

2794

ratings_df_s['title'].unique()

array(['Rocky III', 'Greed', 'American Pie', ..., 'K-PAX',
       'Starship Troopers', 'Spirited Away'], dtype=object)

ratings_df_s.groupby([ratings_df_s['title'],ratings_df_s['rating']]).count().reset_index()

ratings_df_s.groupby(ratings_df_s['title']).count().reset_index()

ratings_df_s_allcounts = ratings_df_s.groupby(ratings_df_s['title'])['userId'].count().reset_index()

ratings_df_s_allcounts = ratings_df_s_allcounts.rename(columns = {'userId':'totalRatings'})

ratings_df_s_allcounts

ratings_df_s_allcounts.shape

(2794, 2)

ratings_df_s_allcounts['totalRatings'].describe()

count    2794.000000
mean       16.102004
std        31.481795
min         1.000000
25%         1.000000
50%         4.000000
75%        15.750000
max       324.000000
Name: totalRatings, dtype: float64

ratings_df_s_allcounts.hist()

array([[<AxesSubplot:title={'center':'totalRatings'}>]], dtype=object)

ratings_df_s_allcounts['totalRatings'].quantile(np.arange(0.6,1, 0.01)) #分位点

0.60      7.00
0.61      7.00
0.62      7.00
0.63      8.00
0.64      8.00
0.65      9.00
0.66      9.00
0.67     10.00
0.68     10.00
0.69     11.00
0.70     12.00
0.71     12.00
0.72     13.00
0.73     14.00
0.74     14.00
0.75     15.75
0.76     17.00
0.77     18.00
0.78     19.00
0.79     20.00
0.80     21.00
0.81     22.33
0.82     24.00
0.83     26.00
0.84     27.00
0.85     29.00
0.86     31.00
0.87     34.00
0.88     37.00
0.89     41.77
0.90     45.00
0.91     49.00
0.92     52.56
0.93     59.00
0.94     64.42
0.95     71.00
0.96     83.28
0.97     98.21
0.98    119.14
0.99    168.49
Name: totalRatings, dtype: float64

• 从分位点数据分析可以看出，21%的电影 评分记录数超过20个

votes_count_threshold = 20

ratings_df_s_top=ratings_df_s_allcounts.query('totalRatings > @votes_count_threshold').reset_index()

ratings_df_s_top

ratings_df_s_top.drop(['index'],axis=1,inplace=True)

ratings_df_s_top.head()

ratings_df_s['title']

0                Rocky III
1                Rocky III
2                Rocky III
3                Rocky III
4                Rocky III
               ...        
44984       Love Is a Ball
44985              Solaris
44986                K-PAX
44987    Starship Troopers
44988        Spirited Away
Name: title, Length: 44989, dtype: object

ratings_df_s_top['title']

0      20,000 Leagues Under the Sea
1             2001: A Space Odyssey
2              24 Hour Party People
3                     28 Days Later
4                    28 Weeks Later
                   ...             
575                      Young Adam
576              Young Frankenstein
577              Young and Innocent
578                        Zatoichi
579                             xXx
Name: title, Length: 580, dtype: object

ratings_df_s[ratings_df_s['title'].isin(ratings_df_s_top['title'])]

ratings_df_s[ratings_df_s['title'].isin(ratings_df_s_top['title'])]  #得到评分数量大于20的

ratings_df_s[~ratings_df_s['title'].isin(ratings_df_s_top['title'])] # 得到评分数量小于20的

ratings_df_s_cntD20 = ratings_df_s[ratings_df_s['title'].isin(ratings_df_s_top['title'])]

ratings_df_s_cntX20 = ratings_df_s[~ratings_df_s['title'].isin(ratings_df_s_top['title'])]

ratings_df_s_cntD20.shape

(34552, 4)

ratings_df_s_cntX20.shape

(10437, 4)

ratings_df_s_cntD20.isna().sum() #检查有无缺失值

userId     0
movieId    0
rating     0
title      0
dtype: int64

ratings_df_s_cntD20.duplicated(['userId','title']).sum()

140

ratings_df_s_cntD20=ratings_df_s_cntD20.drop_duplicates(['userId','title']) # 只保留每个用户对每个电影的一条评论记录

ratings_df_s_cntD20

ratings_df_s_cntD20.duplicated(['userId','title']).sum()

0

# 将一个dataframe的记录数据整合成表格，而且是按照pivot(‘index=xx’,’columns=xx’,’values=xx’)来整合的。还有另外一种写法，就是pivot(‘索引列’，‘列名’，‘值’)。
ratings_df_s_cntD20_for_apriori = ratings_df_s_cntD20.pivot(index='userId',columns='title',values='rating')

ratings_df_s_cntD20_for_apriori

ratings_df_s_cntD20_for_apriori= ratings_df_s_cntD20_for_apriori.fillna(0) #缺失值 填充0

def encode_units(x): # 有效评分规则， 1表示有效，0 表示无效
    if x <= 0:
        return 0
    if x>0:
        return 1

ratings_df_s_cntD20_for_apriori = ratings_df_s_cntD20_for_apriori.applymap(encode_units)

计算频繁项集 和关联规则

from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules

ratings_df_s_cntD20_for_apriori.head()

ratings_df_s_cntD20_for_apriori.isna().sum() #检查是否有nan值

title
20,000 Leagues Under the Sea    0
2001: A Space Odyssey           0
24 Hour Party People            0
28 Days Later                   0
28 Weeks Later                  0
                               ..
Young Adam                      0
Young Frankenstein              0
Young and Innocent              0
Zatoichi                        0
xXx                             0
Length: 580, dtype: int64

frequent_itemsets = apriori(ratings_df_s_cntD20_for_apriori, min_support=0.10, use_colnames=True)  #生成符合条件的频繁项集

frequent_itemsets.sort_values('support',ascending=False)  #support降序排列的频繁项集

rules= association_rules(frequent_itemsets, metric="lift", min_threshold=1)  #生成关联规则，只保留lift>1的部分
rules

rules.sort_values('lift',ascending=False)

• 结果说明：上述输出的即为所有关联规则的结果，每一行代表一个关联规则，其中行号1473所在的关联规则(Waiter->Muxmauschenstill)关联度最高(conviction值越大，代表antecedents与consequents的关联度越大）)。

电影推荐

推荐电影列表

all_antecedents = [list(x) for x in rules['antecedents'].values]

desired_indices = [i for i in range(len(all_antecedents)) if len(all_antecedents[i]) == 1 and all_antecedents[i][0] == 'Batman Returns'] 

apriori_recommendations =rules.iloc[desired_indices,].sort_values(by=['lift'],ascending=False)

apriori_recommendations.head()

apriori_recommendations_list = [list(x) for x in apriori_recommendations['consequents'].values]

print("Apriori Recommendations for movie: Batman Returns\n")
for i in range(5):
    print("{0}:{1} with lift of {2}" .format(i+1, apriori_recommendations_list[i], apriori_recommendations.iloc[i,6]))

Apriori Recommendations for movie: Batman Returns

1:['The Hours', 'Monsoon Wedding', 'Silent Hill', 'Reservoir Dogs'] with lift of 3.215208333333333
2:['Reservoir Dogs', 'Wag the Dog', 'Silent Hill'] with lift of 3.2132394366197183
3:['Monsoon Wedding', 'Silent Hill', 'Reservoir Dogs', 'Sissi'] with lift of 3.168611111111111
4:['Monsoon Wedding', 'Silent Hill', 'Reservoir Dogs', 'Rain Man'] with lift of 3.168611111111111
5:['The Hours', 'Reservoir Dogs', 'Silent Hill'] with lift of 3.139935897435898

推荐单部电影

apriori_single_recommendations = apriori_recommendations.iloc[[x for x in range(len(apriori_recommendations_list)) if len(apriori_recommendations_list[x]) ==1],]

apriori_single_recommendations_list = [list(x) for x in apriori_single_recommendations['consequents'].values]

print("Apriori single-movie Recommendations for movie: Batman Returns\n")
for i in range(5):
    print("{0}: {1}, with lift of {2}".format(i+1,apriori_single_recommendations_list[i][0],apriori_single_recommendations.iloc[i,6]))

Apriori single-movie Recommendations for movie: Batman Returns

1: Reservoir Dogs, with lift of 2.6094444444444447
2: Ariel, with lift of 2.5397663551401872
3: Wag the Dog, with lift of 2.496744186046512
4: To Kill a Mockingbird, with lift of 2.478125
5: Romeo + Juliet, with lift of 2.4705000000000004

• 结果说明：我们约束consequents(后件)的长度为1，选出lift降序排列的前五个关联规则(关联规则格式为前件——>后件）。对于用户观看的电影记录《Batman Returns》，即antecedents(前件），我们根据规则按照推荐程度降序给出了单部电影推荐结果

协同过滤

基于user的协同过滤

• 在海量的用户中发现一小部分和你品味比较相近的，在协同过滤中，这些用户称为邻居，然后根据他们喜欢的东西组织成一个排序的目录来推荐给你
  • 重点就是怎样去寻找和你比较相似的用户，怎么将那些邻居的喜好组织成一个排序的目录给用户
    • 在世纪钟给出一个数字K表示和你最为相似的用户。
    • 在计算相似度的时候，理论上要计算被推荐的用户与所有用户的相似度，但是当数据量比较大的时候，这样做是很费时间的 ，
    • 数据集中可能有很多用户和需要被推荐的用户是没有关系的， 在计算是完全是没有必要的，
    • 所以需要物品到用户的反查表，也就是没一件物品对应的用户信息，有了这个表，就可以过滤掉很多和你没有关系的用户，减少计算量。
  • 总结来说，推荐的过程就是先计算用户之间的相似度，根据相似度的高低选取前K个用户，在这K个用户中计算每一件物品的推荐程度。

# 读取ratings_small.csv数据用于建模
ratings_small_path = "./movie_dataset/ratings_small.csv"
ratings_small_df = pd.read_csv(ratings_small_path)

ratings_small_df.shape

(100004, 4)

ratings_small_df.head()

# 原始的movieId 并非从0到1 的连续值， 为方便更贱user-item矩阵， 重新排列movie_id
movie_id = ratings_small_df['movieId'].drop_duplicates()
movie_id = pd.DataFrame(movie_id)
movie_id['movieid'] = range(len(movie_id))

movie_id

ratings_small_df = pd.merge(ratings_small_df, movie_id, on =['movieId'], how='left')
ratings_small_df

ratings_small_df = ratings_small_df[['userId','movieid','rating','timestamp']]  #更新 movieId ----> movieid
ratings_small_df

# 用户物品统计
# unique()是以 数组形式（numpy.ndarray）返回列的所有唯一值（特征的所有唯一值）
# nunique() Return number of unique elements in the object.即返回的是唯一值的个数

n_users = ratings_small_df.userId.nunique()
n_users

671

n_items = ratings_small_df.movieid.nunique()
n_items 

9066

# 拆分数据集
from sklearn.model_selection import train_test_split
#按照训练集70% 测试集30%的比例 对数据进行拆分
train_data,test_data = train_test_split(ratings_small_df,test_size= 0.3)

train_data

# 训练集 用户-物品 矩阵
user_item_matrix = np.zeros((n_users,n_items))
user_item_matrix.shape

(671, 9066)

# iterrows() : 将DataFrame迭代成（index ,series）
# iteritems()： 将DataFrame迭代成（列名，series）
# itertuples()： 将DataFrame迭代成元组 
for line in train_data.itertuples():
    user_item_matrix[line[1]-1,line[2]]=line[3]

user_item_matrix

array([[0., 3., 3., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]])

user_item_matrix.shape

(671, 9066)

# 构建用户相似矩阵 ---采用余弦距离
from sklearn.metrics.pairwise import pairwise_distances
# 相似度计算 定义余弦距离
user_similarity_m = pairwise_distances(user_item_matrix,metric='cosine')  # 每个用户为1行数据，故此处不需要再进行转置

a=[[1,3],[2,2]]
a

[[1, 3], [2, 2]]

pairwise_distances(a,metric='euclidean')

array([[0.        , 1.41421356],
       [1.41421356, 0.        ]])

b = np.array([[1,2],[1,3],[2,1]])
b

array([[1, 2],
       [1, 3],
       [2, 1]])

pairwise_distances(b,metric='euclidean') #结果数组的第一行第二列表示 a[0]与a[1]的距离

array([[0.        , 1.        , 1.41421356],
       [1.        , 0.        , 2.23606798],
       [1.41421356, 2.23606798, 0.        ]])

pairwise_distances(b,metric='cosine')

array([[0.        , 0.01005051, 0.2       ],
       [0.01005051, 0.        , 0.29289322],
       [0.2       , 0.29289322, 0.        ]])

b.shape

(3, 2)

b[1]

array([1, 3])

b[0]

array([1, 2])

user_similarity_m.shape

(671, 671)

user_similarity_m[0:5,0:5].round(2)

array([[0.  , 1.  , 1.  , 0.94, 0.97],
       [1.  , 0.  , 0.89, 0.93, 0.92],
       [1.  , 0.89, 0.  , 0.93, 0.93],
       [0.94, 0.93, 0.93, 0.  , 0.94],
       [0.97, 0.92, 0.93, 0.94, 0.  ]])

user_similarity_m_triu = np.triu(user_similarity_m,k=1) #取得上三角数据
np.round(user_similarity_m_triu[user_similarity_m_triu.nonzero()],3)

array([1.   , 1.   , 0.938, ..., 0.934, 0.919, 0.814])

user_sim_nonzero = np.round(user_similarity_m_triu[user_similarity_m_triu.nonzero()],3)

np.percentile(user_sim_nonzero,np.arange(0,101,10))

array([0.316, 0.844, 0.885, 0.911, 0.93 , 0.947, 0.961, 0.976, 1.   ,
       1.   , 1.   ])

训练集预测

mean_user_rating = user_item_matrix.mean(axis=1)
mean_user_rating

array([0.00297816, 0.0198544 , 0.01301566, 0.06265167, 0.03027796,
       0.01196779, 0.02404589, 0.03805427, 0.01114053, 0.0147805 ,
       0.01047871, 0.01301566, 0.01615928, 0.004743  , 0.33984116,
       0.01069932, 0.0991617 , 0.0147805 , 0.11780278, 0.02294286,
       0.0443415 , 0.04936025, 0.21111846, 0.00683874, 0.00617692,
       0.04031546, 0.00816236, 0.01555261, 0.00363997, 0.29643724,
       0.02625193, 0.01080962, 0.03684094, 0.05702625, 0.0025921 ,
       0.03000221, 0.01147143, 0.03838518, 0.0196338 , 0.01301566,
       0.0592323 , 0.0196338 , 0.02172954, 0.00694904, 0.00512905,
       0.01312597, 0.01069932, 0.14212442, 0.02371498, 0.01169204,
       0.01069932, 0.01941319, 0.00893448, 0.01384293, 0.00838297,
       0.14615045, 0.06254136, 0.01753805, 0.02090227, 0.02018531,
       0.04197   , 0.0172623 , 0.02454225, 0.00739025, 0.00694904,
       0.01544231, 0.02856828, 0.03331127, 0.02327377, 0.02856828,
       0.00794176, 0.05035297, 0.42096845, 0.01544231, 0.0394882 ,
       0.00573572, 0.07346128, 0.08234061, 0.00921024, 0.0100375 ,
       0.05283477, 0.01235385, 0.04555482, 0.03424884, 0.0247077 ,
       0.05084933, 0.00650783, 0.06281712, 0.02448709, 0.01577322,
       0.04671299, 0.02779616, 0.0444518 , 0.0497463 , 0.08769027,
       0.02288771, 0.03160159, 0.02332892, 0.0497463 , 0.00661813,
       0.0173726 , 0.1978822 , 0.02437679, 0.02360468, 0.13942202,
       0.01411869, 0.00656298, 0.00783146, 0.00628723, 0.04015001,
       0.09541143, 0.00650783, 0.00959629, 0.00827267, 0.01351202,
       0.00667328, 0.01389808, 0.05592323, 0.17185087, 0.03833002,
       0.02503861, 0.00882418, 0.00937569, 0.02415619, 0.0666777 ,
       0.01808957, 0.00573572, 0.09695566, 0.00595632, 0.09254357,
       0.01433929, 0.0297816 , 0.03518641, 0.10065078, 0.00529451,
       0.01637988, 0.02018531, 0.02024046, 0.02090227, 0.01213325,
       0.00816236, 0.01103022, 0.02365983, 0.01158173, 0.01455989,
       0.02134348, 0.01312597, 0.04081182, 0.06121774, 0.09557688,
       0.01654533, 0.05664019, 0.01698654, 0.00926539, 0.01279506,
       0.01604897, 0.08697331, 0.00562541, 0.04070152, 0.03132583,
       0.02702405, 0.00915508, 0.02007501, 0.02421134, 0.10870285,
       0.01422899, 0.00761085, 0.02790646, 0.03623428, 0.00590117,
       0.01588352, 0.0051842 , 0.01169204, 0.00639753, 0.03551732,
       0.05834988, 0.07059343, 0.03259431, 0.01080962, 0.00452239,
       0.01108537, 0.04301787, 0.01433929, 0.01125083, 0.05625414,
       0.0099272 , 0.1014229 , 0.02867858, 0.03320097, 0.02150893,
       0.00739025, 0.01687624, 0.01886168, 0.01367748, 0.10533863,
       0.02702405, 0.02051621, 0.01979925, 0.12221487, 0.05846018,
       0.03910214, 0.01831017, 0.01086477, 0.00871388, 0.05542687,
       0.00948599, 0.00672844, 0.01604897, 0.00452239, 0.00683874,
       0.01610413, 0.2176263 , 0.18409442, 0.06254136, 0.01753805,
       0.02465255, 0.03430399, 0.01433929, 0.03855063, 0.07842488,
       0.00501875, 0.02658284, 0.01158173, 0.02625193, 0.00827267,
       0.00921024, 0.00750055, 0.02349437, 0.00987205, 0.03347673,
       0.00937569, 0.20416942, 0.00871388, 0.03160159, 0.04858813,
       0.06507831, 0.01075447, 0.02432164, 0.0843812 , 0.07246856,
       0.0147805 , 0.14328259, 0.08151335, 0.02195014, 0.04268696,
       0.00739025, 0.07677035, 0.03595853, 0.0051842 , 0.05581293,
       0.03628943, 0.01147143, 0.05840503, 0.03739246, 0.04252151,
       0.00650783, 0.02768586, 0.01025811, 0.00926539, 0.01235385,
       0.01047871, 0.13710567, 0.03000221, 0.01091992, 0.054379  ,
       0.00959629, 0.01158173, 0.10324289, 0.00628723, 0.06176925,
       0.02029561, 0.01158173, 0.0296713 , 0.01114053, 0.06585043,
       0.00479815, 0.01808957, 0.0147805 , 0.01136113, 0.00838297,
       0.02509376, 0.03524156, 0.05333113, 0.01433929, 0.10302228,
       0.00915508, 0.0893448 , 0.02029561, 0.0049636 , 0.02090227,
       0.02553497, 0.08018972, 0.02217075, 0.25672844, 0.06849768,
       0.00634238, 0.03662034, 0.02647253, 0.11763733, 0.01389808,
       0.00551511, 0.00750055, 0.06243106, 0.03309067, 0.00595632,
       0.16997573, 0.02029561, 0.0148908 , 0.04594088, 0.00468784,
       0.23830796, 0.07290977, 0.08112729, 0.01169204, 0.01246415,
       0.03524156, 0.00573572, 0.01588352, 0.00595632, 0.01571807,
       0.02283256, 0.01323627, 0.00700419, 0.04180454, 0.00446724,
       0.00783146, 0.02073682, 0.04649239, 0.00584602, 0.02680344,
       0.00689389, 0.00816236, 0.02503861, 0.01086477, 0.007666  ,
       0.00816236, 0.00330907, 0.01323627, 0.02950585, 0.01384293,
       0.00595632, 0.05868079, 0.01114053, 0.04500331, 0.0619347 ,
       0.09055813, 0.00650783, 0.01621443, 0.00639753, 0.0495257 ,
       0.01378778, 0.02443194, 0.1039047 , 0.01544231, 0.09039268,
       0.00419148, 0.00948599, 0.15243768, 0.01483565, 0.0098169 ,
       0.01533201, 0.03071917, 0.05404809, 0.00909993, 0.0224465 ,
       0.0097066 , 0.05217295, 0.00628723, 0.01345687, 0.03055372,
       0.0446724 , 0.00849327, 0.06165895, 0.00838297, 0.00705934,
       0.01808957, 0.00645268, 0.03750276, 0.01990955, 0.28375248,
       0.02945069, 0.07654975, 0.01544231, 0.11973307, 0.03132583,
       0.02691374, 0.09276417, 0.22865652, 0.01246415, 0.03430399,
       0.02923009, 0.00617692, 0.0125193 , 0.04511361, 0.00683874,
       0.03540702, 0.01632473, 0.01544231, 0.00595632, 0.01676594,
       0.024818  , 0.09303993, 0.00783146, 0.0098169 , 0.11675491,
       0.0270792 , 0.10699316, 0.05978381, 0.01566292, 0.00799691,
       0.00882418, 0.05129054, 0.00650783, 0.01698654, 0.00893448,
       0.02724465, 0.04114273, 0.0494154 , 0.01643503, 0.02823737,
       0.0101478 , 0.0296713 , 0.09458416, 0.00799691, 0.01588352,
       0.06507831, 0.09458416, 0.04560997, 0.00457754, 0.09618354,
       0.09303993, 0.02013016, 0.06221046, 0.05382749, 0.00606662,
       0.02161924, 0.00683874, 0.00612177, 0.05779837, 0.01367748,
       0.03568277, 0.07572248, 0.01775866, 0.00441209, 0.00540481,
       0.00904478, 0.01808957, 0.00639753, 0.00871388, 0.03943305,
       0.01599382, 0.33085153, 0.02294286, 0.0101478 , 0.00821752,
       0.01660049, 0.14179351, 0.02272226, 0.00705934, 0.08283697,
       0.15784249, 0.0121884 , 0.13335539, 0.01058901, 0.01119568,
       0.0593426 , 0.02095742, 0.30228326, 0.0048533 , 0.01869623,
       0.0569711 , 0.24652548, 0.02614163, 0.01301566, 0.14284139,
       0.01114053, 0.00490845, 0.02774101, 0.03132583, 0.1185749 ,
       0.1435032 , 0.01819987, 0.03259431, 0.00573572, 0.004743  ,
       0.0398191 , 0.04037062, 0.01781381, 0.00672844, 0.0051842 ,
       0.01875138, 0.01941319, 0.02923009, 0.02415619, 0.00617692,
       0.03309067, 0.03419369, 0.0048533 , 0.01235385, 0.05741231,
       0.05658504, 0.03353188, 0.01334657, 0.004743  , 0.09927201,
       0.0051842 , 0.01125083, 0.01334657, 0.2351092 , 0.04367968,
       0.00948599, 0.00921024, 0.00584602, 0.1037944 , 0.00876903,
       0.03805427, 0.01411869, 0.19170527, 0.05619899, 0.03987426,
       0.01384293, 0.06083168, 0.04003971, 0.01968895, 0.03992941,
       0.00777631, 0.03171189, 0.03325612, 0.16804544, 0.02062652,
       0.03298037, 0.01384293, 0.0394882 , 0.08030002, 0.01378778,
       0.03011251, 0.10070593, 0.00739025, 0.01058901, 0.00551511,
       0.00683874, 0.01704169, 0.01544231, 0.09265387, 0.02713435,
       0.02178469, 0.63484447, 0.03562762, 0.00623208, 0.03353188,
       0.02360468, 0.00783146, 0.06358923, 0.01511141, 0.01831017,
       0.00959629, 0.01329142, 0.07224796, 0.04378998, 0.03253916,
       0.07798368, 0.07026252, 0.04616148, 0.52404589, 0.00871388,
       0.00777631, 0.01147143, 0.01180234, 0.02283256, 0.03634458,
       0.01577322, 0.02950585, 0.0101478 , 0.09022722, 0.14284139,
       0.01125083, 0.0917163 , 0.00805206, 0.00209574, 0.22887712,
       0.00595632, 0.03502096, 0.00821752, 0.06072138, 0.09728657,
       0.0150011 , 0.15938672, 0.01400838, 0.01047871, 0.02228105,
       0.00849327, 0.03904699, 0.02128833, 0.02514891, 0.05118023,
       0.14399956, 0.06243106, 0.07842488, 0.05757776, 0.01119568,
       0.01268476, 0.03926759, 0.03617913, 0.00330907, 0.11096404,
       0.0196338 , 0.12618575, 0.08879329, 0.02283256, 0.01913744,
       0.01080962, 0.01742775, 0.01560777, 0.02889918, 0.10225017,
       0.01069932, 0.01764836, 0.0100375 , 0.01257445, 0.04086698,
       0.02614163, 0.01185749, 0.03105008, 0.39383411, 0.02079197,
       0.04290757, 0.04500331, 0.0223362 , 0.00959629, 0.0075557 ,
       0.00937569, 0.01185749, 0.00772116, 0.00534966, 0.00750055,
       0.00739025, 0.00976175, 0.004743  , 0.01455989, 0.01191264,
       0.04059122, 0.01169204, 0.00490845, 0.01125083, 0.007666  ,
       0.05834988, 0.05162144, 0.07715641, 0.0245974 , 0.00827267,
       0.00595632, 0.08509817, 0.01753805, 0.20257004, 0.03353188,
       0.0445621 , 0.00419148, 0.01952349, 0.03827487, 0.02950585,
       0.00843812, 0.01742775, 0.00871388, 0.15927642, 0.1088683 ,
       0.00816236, 0.01687624, 0.00739025, 0.0098169 , 0.00716964,
       0.0347452 ])

rating_diff = (user_item_matrix - mean_user_rating[:,np.newaxis])   # np.newaxis作用：为mean_user_rating增加一个维度，实现加减操作
rating_diff

array([[-2.97816016e-03,  2.99702184e+00,  2.99702184e+00, ...,
        -2.97816016e-03, -2.97816016e-03, -2.97816016e-03],
       [-1.98544011e-02, -1.98544011e-02, -1.98544011e-02, ...,
        -1.98544011e-02, -1.98544011e-02, -1.98544011e-02],
       [-1.30156629e-02, -1.30156629e-02, -1.30156629e-02, ...,
        -1.30156629e-02, -1.30156629e-02, -1.30156629e-02],
       ...,
       [-9.81689830e-03, -9.81689830e-03, -9.81689830e-03, ...,
        -9.81689830e-03, -9.81689830e-03, -9.81689830e-03],
       [-7.16964483e-03, -7.16964483e-03, -7.16964483e-03, ...,
        -7.16964483e-03, -7.16964483e-03, -7.16964483e-03],
       [-3.47452019e-02, -3.47452019e-02, -3.47452019e-02, ...,
        -3.47452019e-02, -3.47452019e-02, -3.47452019e-02]])

user_prediction = mean_user_rating[:,np.newaxis] + user_similarity_m.dot(rating_diff) / np.array([np.abs(user_similarity_m).sum(axis=1)]).T
# 处以np.array([np.abs(item_similarity_m).sum(axis=1)]是为了可以使评分在1~5之间，使1~5的标准化

user_prediction

array([[ 8.48587738e-02,  1.11549860e-01,  7.78496257e-02, ...,
        -3.30873704e-02, -3.59785123e-02, -3.59132569e-02],
       [ 9.36489784e-02,  1.35396758e-01,  1.04357090e-01, ...,
        -1.62815182e-02, -1.93136443e-02, -1.93247190e-02],
       [ 9.44428457e-02,  1.33314515e-01,  9.83052575e-02, ...,
        -2.28228892e-02, -2.58037344e-02, -2.59258365e-02],
       ...,
       [ 9.29750987e-02,  1.27902780e-01,  9.32275326e-02, ...,
        -2.60694824e-02, -2.89101875e-02, -2.87905826e-02],
       [ 8.62056229e-02,  1.26697599e-01,  9.17810994e-02, ...,
        -2.88942031e-02, -3.19119828e-02, -3.20590645e-02],
       [ 1.17342284e-01,  1.50739909e-01,  1.17908253e-01, ...,
        -7.69495365e-05, -2.99819315e-03, -3.02101562e-03]])

# 只取数据集中有评分的数据集进行评估
from sklearn.metrics import mean_squared_error
from math import sqrt

prediction_flatten = user_prediction[user_item_matrix.nonzero()]
prediction_flatten

array([0.11154986, 0.07784963, 0.14877094, ..., 0.04236321, 0.01114962,
       0.02448394])

user_item_matrix_flatten = user_item_matrix[user_item_matrix.nonzero()]
user_item_matrix_flatten

array([3., 3., 2., ..., 4., 4., 4.])

error_test = sqrt(mean_squared_error(prediction_flatten,user_item_matrix_flatten)) # 均方根误差计算

error_test

3.390138302832629