I am using delta live table and pub sub to ingest message from 30 different topics in parallel. I noticed that initialization time can be very long around 15 minutes. Does someone knows how to reduced initialization time in dlt ? Thanks You
Hi Oliver,see below screenshot below. first the cluster is initializing. (so the startup is finished the delay does not come from cluster startup)then flow are appended to pipeline (around 30 flows).these flows takes time to get initialize, and i bel...
Hello Louis,As i mentionned gcp serverless is not available in my region europe-west1https://docs.databricks.com/gcp/en/resources/feature-region-supportMy performance issue is not related to cluster startup delay. It is related to initialization of d...
Hi @RiyazAliM Are you using Azure Databricks ?I am using Google Pub / Sub and GCP Databricks. Serverless is still not available in GCP at least not in my region. I think my issue is related to Google Pub Sub who can not initialize connection with DLT...
