Databricks Community

Thanks both. Notebooks are a little too intimidating for some users so we are trying to make them look and feel a bit more like what they are used to. Ideally we would build an app but apps aren't available in our area yet so we need an interim solut...

Hey, I am struck with the same issue, did you find any workaround. I am trying to run DESCRIBE & ALTER command using IDENTIFIER() using databricks-sql-connector. Did u figure out how to run the identifier command statements??

Hi @lprevost  The WorkspaceClient provides APIs to manage Query objects. But it doesn't provide the API to run it. If you need to run the query from a notebook, you can pass the query text into `spark.sql`. It returns SparkDataFrame. I hope this help...

I would expect both the Python process on the driver and Spark's JVM to release memory once you are done with each chunk of data. Otherwise, this sounds like a memory leak. If you suspect this is a problem in the JVM, you can look at heap dumps - the...

@knutasmI have build the solution for it. The way to create DLT pipeline using SPN is to write a code wherein via databricks API a new DLT pipeline will be created and you mentioned owner as a service principal in the API code as shown below. Below m...

Cluster running in shared mode with Databricks runtime 15.4 LTSUC setup as per Databricks guide. I can see system tables everything but data is not populating in the table_lineage table. EventHub Firewall 9093 port is also open.Enabled runtime settin...

Hi @Anish_2 How are you doing today? I agree with @KaranamS's answer.Databricks marks the table as inactive instead of removing it to prevent accidental data loss, allowing you to restore it if needed. Once inactive, the table remains in Unity Catalo...

Hi,I am also facing same issue, is there any ETA to fix it?

Hey! I'm getting this same error connecting to s3 compatible storage - MinIO. Were you able to resolve this? Thanks!

Thanks for the quick response.Your suggestion is essentially what we are currently doing. We have a config file for each environment that is used to create all the widgets. In the first cell of our notebooks, we call a script to create the widgets us...

hi @jhonm_839 thanks for your response to the post. I appreciate your time.I have watermark on dataframe in palce and used last(), min() etc.. aggregate functions its still filing.final_df = df.withWatermark("kafka_ts", "1 hour").withColumn("s_featur...

Hi David,I can't speak exactly to how Poetry handles the dependency resolution of libraries that are already installed, or how that interacts with the Databricks runtime. However, I can offer you some advice on how my team handles this situtation.It'...

I can indeed confirm that adding some additional table properties to the @Dlt attribute in the DLT pipeline definition resolved the earlier issues. Thanks for pointing this out.