Does anyone use Delta Live Table on large scale in production pipelines ? Are they satisfied with the product ?
Recently, Iยดve started a PoC to evaluate the DLT and notice some concerns:
- Excessive use of compute resources when you check the cluster metrics, itยดs necessary a big machine.
- Does not accept single node compute;
- Poor performance to initialize the pipeline, around 30 minutes to mount the graph and start to running
- The fact we canยดt define the specific DBR is worrying, we donยดt have control over what Databricks is changing and the pipeline can break at any time.
I started the use now, but the first impressions were not good. I expected more.
