Delta Live Tables: large use

[Taja]

Does anyone use Delta Live Table on large scale in production pipelines ? Are they satisfied with the product ?

Recently, I´ve started a PoC to evaluate the DLT and notice some concerns:
- Excessive use of compute resources when you check the cluster metrics, it´s necessary a big machine.
- Does not accept single node compute;
- Poor performance to initialize the pipeline, around 30 minutes to mount the graph and start to running
- The fact we can´t define the specific DBR is worrying, we don´t have control over what Databricks is changing and the pipeline can break at any time.

I started the use now, but the first impressions were not good. I expected more.

[RiyazAli]

Hi @Taja,

I agree that DLT pipelines doesn't accept a single node cluster to begin with but you can always choose the instance type for both your driver and the worker nodes.

As far as `waiting for resources` time is concerned, I've seen that DLT takes ~5 mins to spin up the clusters for the first time the resources are being requested, and then this cluster stays active till 2 hours in Dev mode so that you need not have to request for resources again. If it's taking more time than this, I suspect it's the cloud service provider having a cap on the resources. 

Once DLT acquired all the resources, it enters initialising mode where it checks for the execution flow, syntax and once everything is good, it sets the tables up. 

Agree that you can specify a DBR version you want to use, though you can toggle between current channel or preview channel (which contains features which might not be GA yet).

Cheers!