我有这些函数来处理一个 2GB 的文本文件。我将它分成 6 个部分进行同时处理，但它仍然需要 4 个多小时。

还有什么我可以尝试使脚本更快？

一些细节:

1. 我将输入的 csv 送入 while 循环以逐行读取。
2. 我从 read2col 函数的 4 个字段中获取了 csv 行的值
3. 我的 mainf 函数中的 awk 从 read2col 中获取值并进行一些算术计算。我将结果四舍五入到小数点后两位。然后，将该行打印到文本文件。

示例数据:

"111","2018-08-24","01:21","ZZ","AAA","BBB","0","","","ZZ","ZZ111","ZZ110","2018-10-12","07:00","2018-10-12","08:05","2018-10-19","06:30","2018-10-19","09:35","ZZZZ","ZZZZ","A","B","146.00","222.26","76.26","EEE","abc","100.50","45.50","0","E","ESSENTIAL","ESSENTIAL","4","4","7","125","125"

脚本:

read2col()
{
is_one_way=$(echo "$line"| awk -F'","' '{print $7}')
price_outbound=$(echo "$line"| awk -F'","' '{print $30}')
price_exc=$(echo "$line"| awk -F'","' '{print $25}')
tax=$(echo "$line"| awk -F'","' '{print $27}')
price_inc=$(echo "$line"| awk -F'","' '{print $26}')
}


#################################################
#for each line in the csv
mainf()
{
cd $infarepath

while read -r line; do
        #read the value of csv fields into variables
        read2col

        if [[ $is_one_way == 0 ]]; then
                if [[ $price_outbound > 0 ]]; then
                        #calculate price inc and print the entire line to txt file
                        echo $line | awk -v CONVFMT='%.2f' -v pout=$price_outbound -v tax=$tax -F'","' 'BEGIN {OFS = FS} {$25=pout;$26=(pout+(tax / 2)); print}' >>"$csvsplitfile".tmp
                else
                        #divide price ecx and inc by 2 if price outbound is not greater than 0
                        echo $line | awk -v CONVFMT='%.2f' -v pexc=$price_exc -v pinc=$price_inc -F'","' 'BEGIN {OFS = FS} {$25=(pexc / 2);$26=(pinc /2); print}' >>"$csvsplitfile".tmp
                fi
        else
                echo $line >>"$csvsplitfile".tmp
        fi

done < $csvsplitfile
}

最佳答案

您应该做的第一事情是停止为每一行输入调用六个 子shell 来运行awk。让我们进行一些粗略的快速计算。

假设您的输入行大约有 292 个字符(按照您的示例)，一个 2G 的文件将包含超过 730 万行。这意味着您正在启动和停止数量惊人的四千四百万进程。

而且，虽然 Linux 令人钦佩地尽可能高效地处理 fork 和 exec，但这并非没有代价:

pax$ time for i in {1..44000000} ; do true ; done
real 1m0.946s

此外，bash 并没有真正针对这种处理进行优化，它的设计导致了这种特定用例的次优行为。有关详细信息，请参阅 this excellent answer在我们的姐妹网站之一上。

两种文件处理方法的分析(一个程序读取整个文件(每行只有hello)，和bash读取一行在一个时间)如下所示。用于获取时间的两个命令是:

time ( cat somefile >/dev/null )
time ( while read -r x ; do echo $x >/dev/null ; done <somefile )

对于不同的文件大小(user+sys 时间，几次运行的平均值)，这很有趣:

# of lines   cat-method   while-method
----------   ----------   ------------
     1,000       0.375s         0.031s
    10,000       0.391s         0.234s
   100,000       0.406s         1.994s
 1,000,000       0.391s        19.844s
10,000,000       0.375s       205.583s
44,000,000       0.453s       889.402s

由此看来，while 方法似乎可以适用于较小的数据集，但它确实不能很好地扩展。

由于 awk 本身有方法进行计算和格式化输出，因此使用一个单个 awk 脚本处理文件，而不是您的 bash/multi-awk-per-line 组合，将使创建所有这些进程的成本和基于行的延迟消失。

这个脚本将是一个很好的尝试，我们称它为 prog.awk:

BEGIN {
    FMT = "%.2f"
    OFS = FS
}
{
    isOneWay=$7
    priceOutbound=$30
    priceExc=$25
    tax=$27
    priceInc=$26

    if (isOneWay == 0) {
        if (priceOutbound > 0) {
            $25 = sprintf(FMT, priceOutbound)
            $26 = sprintf(FMT, priceOutbound + tax / 2)
        } else {
            $25 = sprintf(FMT, priceExc / 2)
            $26 = sprintf(FMT, priceInc / 2)
        }
    }
    print
}

您只需运行那个单个 awk 脚本:

awk -F'","' -f prog.awk data.txt

根据您提供的测试数据，这里是之前和之后，带有字段编号 25 和 26 的标记:

                                                                                                                                                                                      <-25->   <-26->
"111","2018-08-24","01:21","ZZ","AAA","BBB","0","","","ZZ","ZZ111","ZZ110","2018-10-12","07:00","2018-10-12","08:05","2018-10-19","06:30","2018-10-19","09:35","ZZZZ","ZZZZ","A","B","146.00","222.26","76.26","EEE","abc","100.50","45.50","0","E","ESSENTIAL","ESSENTIAL","4","4","7","125","125"
"111","2018-08-24","01:21","ZZ","AAA","BBB","0","","","ZZ","ZZ111","ZZ110","2018-10-12","07:00","2018-10-12","08:05","2018-10-19","06:30","2018-10-19","09:35","ZZZZ","ZZZZ","A","B","100.50","138.63","76.26","EEE","abc","100.50","45.50","0","E","ESSENTIAL","ESSENTIAL","4","4","7","125","125"

关于linux - 优化大文件的 awk 命令，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/52013810/