几个月来，我在学习 Pandas 的过程中一直试图找出这个问题的答案。我在日常工作中使用 SAS，它的核心外支持非常棒。然而，由于许多其他原因，SAS 作为一款软件很糟糕。

有一天，我希望用 Python 和 Pandas 取代我对 SAS 的使用，但我目前缺乏用于大型数据集的核心外工作流程。我说的不是需要分布式网络的“大数据”，而是大到无法放入内存但小到可以放入硬盘的文件。

我的第一个想法是使用 HDFStore将大型数据集保存在磁盘上并仅将我需要的部分提取到数据帧中进行分析。其他人提到 MongoDB 是一种更易于使用的替代方案。我的问题是这样的:

完成以下工作的最佳实践工作流程有哪些:

最佳答案

我经常以这种方式使用数十 GB 的数据
例如我在磁盘上有通过查询读取的表，创建数据并附加回来。

值得一读the docs和 late in this thread有关如何存储数据的一些建议。

将影响您存储数据方式的详细信息，例如:
提供尽可能多的细节；我可以帮你开发一个结构。

import numpy as np
import pandas as pd

# create a store
store = pd.HDFStore('mystore.h5')

# this is the key to your storage:
#    this maps your fields to a specific group, and defines 
#    what you want to have as data_columns.
#    you might want to create a nice class wrapping this
#    (as you will want to have this map and its inversion)  
group_map = dict(
    A = dict(fields = ['field_1','field_2',.....], dc = ['field_1',....,'field_5']),
    B = dict(fields = ['field_10',......        ], dc = ['field_10']),
    .....
    REPORTING_ONLY = dict(fields = ['field_1000','field_1001',...], dc = []),

)

group_map_inverted = dict()
for g, v in group_map.items():
    group_map_inverted.update(dict([ (f,g) for f in v['fields'] ]))

for f in files:
   # read in the file, additional options may be necessary here
   # the chunksize is not strictly necessary, you may be able to slurp each 
   # file into memory in which case just eliminate this part of the loop 
   # (you can also change chunksize if necessary)
   for chunk in pd.read_table(f, chunksize=50000):
       # we are going to append to each table by group
       # we are not going to create indexes at this time
       # but we *ARE* going to create (some) data_columns

       # figure out the field groupings
       for g, v in group_map.items():
             # create the frame for this group
             frame = chunk.reindex(columns = v['fields'], copy = False)    

             # append it
             store.append(g, frame, index=False, data_columns = v['dc'])

frame = store.select(group_that_I_want)
# you can optionally specify:
# columns = a list of the columns IN THAT GROUP (if you wanted to
#     select only say 3 out of the 20 columns in this sub-table)
# and a where clause if you want a subset of the rows

# do calculations on this frame
new_frame = cool_function_on_frame(frame)

# to 'add columns', create a new group (you probably want to
# limit the columns in this new_group to be only NEW ones
# (e.g. so you don't overlap from the other tables)
# add this info to the group_map
store.append(new_group, new_frame.reindex(columns = new_columns_created, copy = False), data_columns = new_columns_created)

# This may be a bit tricky; and depends what you are actually doing.
# I may need to modify this function to be a bit more general:
report_data = store.select_as_multiple([groups_1,groups_2,.....], where =['field_1>0', 'field_1000=foo'], selector = group_1)

store.select(group, where = ['field_1000=foo', 'field_1001>0'])

关于python - "Large data"使用 Pandas 的工作流程，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/14262433/