cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

gustavomcarmo-h
by New Contributor III
  • 1479 Views
  • 5 replies
  • 2 kudos

Resolved! Is there a way to list the dlt maintenance jobs through the API?

After creating the delta pipeline, I would like to get details from the dlt maintenance job automatically created by Databricks, like the scheduled time when the dlt maintenance tasks will be executed. However, it seems the Job API 2.1 doesn't cover ...

  • 1479 Views
  • 5 replies
  • 2 kudos
Latest Reply
gustavomcarmo-h
New Contributor III
  • 2 kudos

Hi @Debayan Mukherjee​ ,Actually the Databricks Jobs API documentation has not been fixed yet. The parameter `job_type` should be included in the list endpoint request documentation. Please do this in order to avoid unnecessary questions here in the ...

  • 2 kudos
4 More Replies
User16783853501
by New Contributor II
  • 1096 Views
  • 3 replies
  • 0 kudos

best practice for optimizedWrites and Optimize

What is the best practice for a delta pipeline with very high throughput to avoid small files problem and also reduce the need for external OPTIMIZE frequently?  

  • 1096 Views
  • 3 replies
  • 0 kudos
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

The general practice in use is to enable only optimize writes and disable auto-compaction. This is because the optimize writes will introduce an extra shuffle step which will increase the latency of the write operation. In addition to that, the auto-...

  • 0 kudos
2 More Replies
Anand_Ladda
by Honored Contributor II
  • 783 Views
  • 1 replies
  • 1 kudos
  • 783 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anand_Ladda
Honored Contributor II
  • 1 kudos

Notebooks with Delta Live Table/View definition just contain the pipeline definition. In order to execute Delta Live Tables Notebooks you need to define a Pipeline via the Jobs UI. Pipeline carries with it the logic to build the dependency graph betw...

  • 1 kudos
Labels