topic How to get the modification time of files from a notebook command? in Data Engineering https://community.databricks.com/t5/data-engineering/how-to-get-the-modification-time-of-files-from-a-notebook/m-p/21026#M14264 Wed, 23 Jun 2021 23:51:48 GMT brickster_2018 2021-06-23T23:51:48Z How to get the modification time of files from a notebook command? https://community.databricks.com/t5/data-engineering/how-to-get-the-modification-time-of-files-from-a-notebook/m-p/21026#M14264 Wed, 23 Jun 2021 23:51:48 GMT https://community.databricks.com/t5/data-engineering/how-to-get-the-modification-time-of-files-from-a-notebook/m-p/21026#M14264 brickster_2018 2021-06-23T23:51:48Z Re: How to get the modification time of files from a notebook command? https://community.databricks.com/t5/data-engineering/how-to-get-the-modification-time-of-files-from-a-notebook/m-p/21027#M14265 <P>The below code snippet is useful to get the modification time of files. </P><PRE><CODE>%scala import scala.util.Try import org.apache.hadoop.conf.Configuration import org.apache.hadoop.fs.{FileSystem, Path} import org.apache.hadoop.io.IOUtils import java.io.IOException val conf = sc.hadoopConfiguration val pathObj=new Path("s3a://my_bucket/location/") val obj= pathObj.getFileSystem(conf) val fs = pathObj.getFileSystem(conf) display(fs.listStatus(pathObj).map(x=&gt;(x.getModificationTime,x.getPath.toString)).toSeq.toDF().withColumn("time",from_unixtime($"_1" /1000)).withColumnRenamed("_2","path").drop("_1"))</CODE></PRE><P></P> Wed, 23 Jun 2021 23:53:07 GMT https://community.databricks.com/t5/data-engineering/how-to-get-the-modification-time-of-files-from-a-notebook/m-p/21027#M14265 brickster_2018 2021-06-23T23:53:07Z