Data Engineering

by Srishti • New Contributor

08-24-2022 5:24:37 AM

1070 Views
0 replies
0 kudos

Want to extract the start time and end time of tasks under particular job.

I have one job that reran multiple times which took 101 hours. Ideally the execution of that job is 8 hours. Using jobs 2.1 API I am able to extract the start and end time of job ID and run ID. This only helps me to get the duration of 101 hours , bu...

Data Engineering

1070 Views
0 replies
0 kudos

08-24-2022 5:24:37 AM

by brickster_2018 • Databricks Employee

06-25-2021 9:47:04 AM

2458 Views
1 replies
0 kudos

Resolved! Can auto-loader process the overwritten file

I have a directory where I get files with the same multiple times. Will Auto-loader process all the files or will it process the first and ignore the rest

Data Engineering

2458 Views
1 replies
0 kudos

06-25-2021 9:47:04 AM

View Replies

Latest Reply

brickster_2018
Databricks Employee

06-25-2021 9:47:20 AM

0 kudos

Autoloader has an option - "cloudFiles. allowOverwrites". This determines whether to allow input directory file changes to overwrite existing data. This option is available in Databricks Runtime 7.6 and above.

0 kudos

06-25-2021 9:47:20 AM

Databricks Community

Forum Posts

Want to extract the start time and end time of tasks under particular job.

Resolved! Can auto-loader process the overwritten file