cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
gustavomcarmo-h
by • New Contributor III
  • 1525 Views
  • 5 replies
  • 2 kudos

Resolved! Is there a way to list the dlt maintenance jobs through the API?

After creating the delta pipeline, I would like to get details from the dlt maintenance job automatically created by Databricks, like the scheduled time when the dlt maintenance tasks will be executed. However, it seems the Job API 2.1 doesn't cover ...

Data Engineering
Reply
  • 1525 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
gustavomcarmo-h
New Contributor III
  • 2 kudos

Hi @Debayan Mukherjee​ ,Actually the Databricks Jobs API documentation has not been fixed yet. The parameter `job_type` should be included in the list endpoint request documentation. Please do this in order to avoid unnecessary questions here in the ...

  • 2 kudos
4 More Replies
User16783853501
by • New Contributor II
  • 1125 Views
  • 3 replies
  • 0 kudos

best practice for optimizedWrites and Optimize

What is the best practice for a delta pipeline with very high throughput to avoid small files problem and also reduce the need for external OPTIMIZE frequently?  

Data Engineering
Reply
  • 1125 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

The general practice in use is to enable only optimize writes and disable auto-compaction. This is because the optimize writes will introduce an extra shuffle step which will increase the latency of the write operation. In addition to that, the auto-...

  • 0 kudos
2 More Replies
aladda
by • Honored Contributor II
  • 807 Views
  • 1 replies
  • 1 kudos

Resolved! Why can't I run a notebook that has Delta Live Tables/Views defined against a cluster like other Notebooks

Data Engineering
Reply
  • 807 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
aladda
Honored Contributor II
  • 1 kudos

Notebooks with Delta Live Table/View definition just contain the pipeline definition. In order to execute Delta Live Tables Notebooks you need to define a Pipeline via the Jobs UI. Pipeline carries with it the logic to build the dependency graph betw...

  • 1 kudos
Labels