How to get the modification time of files from a notebook command?

[brickster_2018 · Accepted solution]

The below code snippet is useful to get the modification time of files.

%scala
import scala.util.Try
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.hadoop.io.IOUtils
import java.io.IOException
val conf = sc.hadoopConfiguration
val pathObj=new Path("s3a://my_bucket/location/")
val obj= pathObj.getFileSystem(conf)
val fs = pathObj.getFileSystem(conf)
display(fs.listStatus(pathObj).map(x=>(x.getModificationTime,x.getPath.toString)).toSeq.toDF().withColumn("time",from_unixtime($"_1" /1000)).withColumnRenamed("_2","path").drop("_1"))