Databricks Community

@angelop I tried it as well and encountered the same error. A new collaborator needs to be set up. If that’s not feasible, it would be advisable to reach out to Databricks support.By the way, the following video provides a more detailed explanation a...

In this case, please make sure you specify the schema explicitly when reading the Parquet files and do not specify any inference options. Something like spark.readStream.format("cloudFiles").schema(schema)... If you want to more easily grab the schem...

@Wundermobility  The best way to debug this is to look at the Spark UI to see if a job has been launched. One thing to call out is that trigger.Once is deprecated - we recommend using trigger.availableNow instead to avoid overwhelming the cluster.

HI, @jonathan-dufaul For such reports, I think it would be appropriate to click on your profile icon in the top-right corner of the workspace and use the "Send Feedback" option. 

Hi, @holychs Did you encounter any error messages related to an OOM (Out of Memory) error?It’s possible that the driver node of the cluster doesn’t have sufficient resources (CPU, memory) to handle multiple concurrent jobs.

Hi @Yaadhudbe, We would need to review your DLT setup, cluster settings and spark processing to better understand the OOM errors and possible suggestions to mitigate the issue. I suggest to file a case with us to conduct a proper investigation.  http...

Were you able to find any solution to the problem?I am also having the similar use-case where I need to run multiple run_job_task and everytime it is spinning up new cluster of its own that is defined in child job.I am not able to find any relevant s...

@Walter_Cthank you so much, that worked perfectly

To apply expectations only if a column exists in Delta Live Tables (DLT), you can use the @Dlt.expect decorator conditionally within your Python function. Here is a step-by-step approach to achieve this: Check if the Column Exists: Before applying th...

@smit_tw Have you tried setting it on the "Advanced" tab as my previous suggests?

The AvailableNow trigger processes all available data as a single batch and then stops. This is different from continuous or micro-batch processing where the system continuously checks for new data. When a query starts with the AvailableNow trigger, ...

I would say that your token should be manually refreshed as mentioned in the following statement in docs: Databricks tools and SDKs that implement the Databricks client unified authentication standard will automatically generate, refresh, and use Dat...

I have used the bellow provider to run the query https://registry.terraform.io/providers/hashicorp/http/latest