cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User16783853501
by • Databricks Employee
  • 3247 Views
  • 4 replies
  • 0 kudos

best practice for optimizedWrites and Optimize

What is the best practice for a delta pipeline with very high throughput to avoid small files problem and also reduce the need for external OPTIMIZE frequently?  

Data Engineering
Reply
  • 3247 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
rajkve
New Contributor II
  • 0 kudos

Hi All,Can anyone who has solved this challenge confirm if the below increases write latency and avoids creating smaller file, based a POC I did, I dont see that behaviour replicable, so I am just wondering. Many thanks.  

  • 0 kudos
3 More Replies
gustavomcarmo-h
by • New Contributor III
  • 4654 Views
  • 5 replies
  • 2 kudos

Resolved! Is there a way to list the dlt maintenance jobs through the API?

After creating the delta pipeline, I would like to get details from the dlt maintenance job automatically created by Databricks, like the scheduled time when the dlt maintenance tasks will be executed. However, it seems the Job API 2.1 doesn't cover ...

Data Engineering
Reply
  • 4654 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
gustavomcarmo-h
New Contributor III
  • 2 kudos

Hi @Debayan Mukherjee​ ,Actually the Databricks Jobs API documentation has not been fixed yet. The parameter `job_type` should be included in the list endpoint request documentation. Please do this in order to avoid unnecessary questions here in the ...

  • 2 kudos
4 More Replies
aladda
by • Databricks Employee
  • 1938 Views
  • 1 replies
  • 1 kudos

Resolved! Why can't I run a notebook that has Delta Live Tables/Views defined against a cluster like other Notebooks

Data Engineering
Reply
  • 1938 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 1 kudos

Notebooks with Delta Live Table/View definition just contain the pipeline definition. In order to execute Delta Live Tables Notebooks you need to define a Pipeline via the Jobs UI. Pipeline carries with it the logic to build the dependency graph betw...

  • 1 kudos
Labels