pathGlobFilter is used to only include files with file names matching the pattern. The syntax follows org.apache.hadoop.fs.GlobFilter. It does not change the behavior of partition discovery.

To load files with paths matching a given glob pattern while keeping the behavior of partition discovery, you can use:

val testGlobFilterDF = spark.read.format("parquet")

 .option("pathGlobFilter", "*.parquet") // json file should be filtered out

 .load("examples/src/main/resources/dir1")

testGlobFilterDF.show()

// +-------------+

// | file|

// +-------------+

// |file1.parquet|

// +-------------+