cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

ChristianRRL
by Contributor III
  • 5239 Views
  • 3 replies
  • 3 kudos

Resolved! DLT Job Clusters: Continuous vs Triggered Cluster Start Times

Hi there,I'm curious if anyone is able to definitively help me answer how DLT Job Clusters operate/run.For example, the following is my baseline understanding of DLT Job Clusters. If I run a Triggered DLT Pipeline (e.g. daily) the job cluster takes m...

  • 5239 Views
  • 3 replies
  • 3 kudos
Latest Reply
melbourne
Contributor
  • 3 kudos

Ideally one would expect clusters used for DLT pipeline to terminate after the pipeline execution has finished. However, while running in `development` environment, you'll notice it doesn't terminate on its own, whereas in `production` it terminates ...

  • 3 kudos
2 More Replies
ChristianRRL
by Contributor III
  • 3682 Views
  • 2 replies
  • 1 kudos

DLT Primary Key Deduplication: Expectations vs. Constraints vs. Other?

I'm trying to figure out what's the best way to "de-duplicate" data via DLT. Currently, my only leads are:Manage data quality with Delta Live Tables | Databricks on AWSVia "Drop invalid records"Constraints on Databricks | Databricks on AWSVia "pre-de...

Get Started Discussions
Auto Loader
autoloader
Delta Live Table
Delta Live Table Pipeline
dlt
  • 3682 Views
  • 2 replies
  • 1 kudos
Latest Reply
Palash01
Valued Contributor
  • 1 kudos

Hey @ChristianRRL ,Based on my understanding you want to de-duplicate your data during your DLT pipeline processing unfortunately I was not able to find a solution to this when I ran into this problem due to the native feature limitations.Limitations...

  • 1 kudos
1 More Replies
ChristianRRL
by Contributor III
  • 3660 Views
  • 2 replies
  • 1 kudos

Resolved! DLT Notebook and Pipeline Separation vs Consolidation

Super basic question. For DLT pipelines I see there's an option to add multiple "Paths". Is it generally best practice to completely separate `bronze` from `silver` notebooks? Or is it more recommended to bundle both raw `bronze` and clean `silver` d...

ChristianRRL_1-1705597040187.png
  • 3660 Views
  • 2 replies
  • 1 kudos
Latest Reply
ChristianRRL
Contributor III
  • 1 kudos

This is great! I completely missed the list view before.

  • 1 kudos
1 More Replies
ChristianRRL
by Contributor III
  • 4046 Views
  • 5 replies
  • 2 kudos

DLT Compute Resources - What Compute Is It???

Hi there, I'm wondering if someone can help me understand what compute resources DLT uses? It's not clear to me at all if it uses the last compute cluster I had been working on, or something else entirely.Can someone please help clarify this? 

ChristianRRL_0-1703025826377.png
  • 4046 Views
  • 5 replies
  • 2 kudos
Latest Reply
quakenbush
Contributor
  • 2 kudos

Well, one thing they emphasize in the 'Adavanced Data Engineer' Training is that job-clusters will terminate within 5 minutes after a job is completed. So this could be in support of your theory to lower costs. I think job-cluster are actually design...

  • 2 kudos
4 More Replies
ChristianRRL
by Contributor III
  • 1501 Views
  • 2 replies
  • 0 kudos

Auto Loader Use Case Question - Centralized Dropzone to Bronze?

Good day,I am trying to use Auto Loader (potentially extending into DLT in the future) to easily pull data coming from an external system (currently located in a single location) and organize it and load it respectively. I am struggling quite a bit a...

  • 1501 Views
  • 2 replies
  • 0 kudos
Latest Reply
ChristianRRL
Contributor III
  • 0 kudos

Quick follow-up on this @Retired_mod (or to anyone else in the Databricks multi-verse who is able to help clarify this case).I understand that the proposed solution would work for a "one-to-one" case where many files are landing in a specific dbfs pa...

  • 0 kudos
1 More Replies
vkuznetsov
by New Contributor III
  • 896 Views
  • 1 replies
  • 0 kudos

Problem sharing a streaming table created in Delta Live Table via Delta Sharing

Hi all,I hope you could help me to figure out what I am missing.I'm trying to do a simple thing. To read the data from the data ingestion zone (csv files saved to Azure Storage Account) using the Delta Live Tables pipeline and share the resulting tab...

vkuznetsov_0-1689259588838.png 2023_07_13_16_48_52_Data_Explorer.png
  • 896 Views
  • 1 replies
  • 0 kudos
Latest Reply
vkuznetsov
New Contributor III
  • 0 kudos

Sorry, I think I've created the post in the wrong thread. Created the same post in the Community Cove.

  • 0 kudos
Labels