Databricks Community

shadowinc · 03-03-2025

Background - I created a SQL function with the name schema.function_name, which returns a table, in a notebook, the function works perfectly, however, I want to execute it via API using SQL Endpoint. In API, I got insufficient privileges error, so gr...

shadowinc · 11-26-2024

Hello team, as DataBricks moved away from hive-style partitioning, we can see some 2-letter partition folders created. And I have observed that the vacuum doesn't delete these folders (even though they are empty). Is there any way to delete those usi...

shadowinc · 06-03-2024

We have a special endpoint that grants access to delta tables and we want to know if we can use SQL endpoints as a linked service in ADF.If yes then which ADF-linked service would be suitable for this?Appreciate your support on this.

shadowinc · 05-18-2024

Hi All,We have a table which has an id column generated by uuid(). For ETL we use databricks/spark sql temporary views. we observed strange behavior between databricks sql temp view (create or replace temporary view) and spark sql temp view (df.creat...

shadowinc · 01-31-2025

For me df.withColumn("New_Column", lit(None).cast(StringType())) this didn't work.I used this instead df.withColumn("New_Column", lit(null).cast(StringType))

shadowinc · 11-26-2024

What about ReOrg delta table https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-reorg-tableDoes it help or make sense to add Re-org then Optimize - Vacuum every week?Reorganize a Delta Lake table by rewriting files to purge ...

shadowinc · 11-02-2024

@stefanhieslas11 Thanks for your input. However, are there other approaches to avoid staging databases like Cosmos or Mongo DB as input to web applications and directly fetch from delta tables while maintaining the read performance? Much appreciated.

shadowinc · 08-18-2024

Thanks for the tips. Will try those. Formatting huge queries is not a good experience for me as I have to scroll up and down for any troubleshooting and it is a huge pain tbh. Rather I separate CTEs with a space and in CTE everything is in a line (Ca...

shadowinc · 08-06-2024

@Retired_mod Thanks for the information, however, this query was about the Databricks SQL endpoint (which is like an engine to execute queries in Databricks)

Databricks Community

User Stats

User Activity

Call SQL Function via API

Delete Partition Folders

Databricks SQL endpoint as Linked Service in Azure Data Factory

spark/databricks temporary views and uuid

Re: Differences between lit(None) or lit(None).cast('string')

Re: Optimize and Vacuum - which is the best order of operations?

Re: How to access final delta table for web application or interface.

Re: Top 10 query performance tuning tips for Databricks Serverless SQL

Re: Databricks SQL endpoint as Linked Service in Azure Data Factory