草庐IT

maxPartitionBytes

全部标签

Spark 3.0参数详解之 spark.sql.files.maxPartitionBytes

1、对应源码位置在接口FileScan的partitions方法中org.apache.spark.sql.execution.datasources.v2.FileScan#partitions:2、生效原理2.1、关键方法之partitionsorg.apache.spark.sql.execution.datasources.v2.FileScan#partitions:protecteddefpartitions:Seq[FilePartition]={valselectedPartitions=fileIndex.listFiles(partitionFilters,dataFilt