Databricks Community

maybe you can add some time between uninstalling of libraries and restartuninstall_libraries()time.sleep(60)  # Wait for the libraries to be uninstalledrestart_cluster()

Hello @AngadSingh ,The error message you've shared identifies the issue you're encountering. The error message: Source code has no table to create or update. occurs because your Delta Live Tables (DLT) pipeline does not have any streaming tables defi...

Will the deleted file be completely cleaned-up from storage only after 207 days (retention being 7 and vacuum interval 200 days)? As the default retention period is 7 days, there will not be any files older than 7 days, unless the retention period is...

I'm also trying to use the foreachBatch method of a Spark Streaming DataFrame with databricks-connect. Given that spark connect supported was added to  `foreachBatch` in 3.5.0, I was expecting this to work.Configuration:- DBR 15.4 (Spark 3.5.0)- data...

Production native approach is to maintain your artifacts in central antifactory and use this location in your .yml for libraries. This way you ensure your antifactory are properly versioned , governed and maintaining some sort of standards. I hope wi...

Hi @Fatimah-Tariq ,What about defining your DLT pipeline as below.This way you will create a stream table that reads from your silver, apply all the needed changes and then write back to your silver.%sql -- Read from the streaming table in the silver...

@johnb1 You can achieve this with the code below. Please review.table_name = "table_name" location = "abfss://container@storage-account.dfs.core.windows.net/path/to/table/data/" spark.sql(f"DROP TABLE IF EXISTS {table_name}") dbutils.fs.rm(location...

I followed the syntax in this documentation link https://docs.databricks.com/en/tables/streaming.htmlI get this error if the STREAM keyword is excluded Please add the STREAM keyword to your FROM clause to turn this relation into a streaming query. SQ...