Databricks Community

shadowinc

Hello team, as DataBricks moved away from hive-style partitioning, we can see some 2-letter partition folders created. And I have observed that the vacuum doesn't delete these folders (even though they are empty). Is there any way to delete those usi...

shadowinc · 06-03-2024

We have a special endpoint that grants access to delta tables and we want to know if we can use SQL endpoints as a linked service in ADF.If yes then which ADF-linked service would be suitable for this?Appreciate your support on this.

shadowinc · 05-18-2024

Hi All,We have a table which has an id column generated by uuid(). For ETL we use databricks/spark sql temporary views. we observed strange behavior between databricks sql temp view (create or replace temporary view) and spark sql temp view (df.creat...

shadowinc

What about ReOrg delta table https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-reorg-tableDoes it help or make sense to add Re-org then Optimize - Vacuum every week?Reorganize a Delta Lake table by rewriting files to purge ...

shadowinc · 11-02-2024

@stefanhieslas11 Thanks for your input. However, are there other approaches to avoid staging databases like Cosmos or Mongo DB as input to web applications and directly fetch from delta tables while maintaining the read performance? Much appreciated.

shadowinc · 08-18-2024

Thanks for the tips. Will try those. Formatting huge queries is not a good experience for me as I have to scroll up and down for any troubleshooting and it is a huge pain tbh. Rather I separate CTEs with a space and in CTE everything is in a line (Ca...

shadowinc · 08-06-2024

@Retired_mod Thanks for the information, however, this query was about the Databricks SQL endpoint (which is like an engine to execute queries in Databricks)

shadowinc · 05-20-2024

Thanks, @Retired_mod I suspected that, but could not find any links for confirming it.

Databricks Community

User Stats

User Activity

Delete Partition Folders

Databricks SQL endpoint as Linked Service in Azure Data Factory

spark/databricks temporary views and uuid

Re: Optimize and Vacuum - which is the best order of operations?

Re: How to access final delta table for web application or interface.

Re: Top 10 query performance tuning tips for Databricks Serverless SQL

Re: Databricks SQL endpoint as Linked Service in Azure Data Factory

Re: spark/databricks temporary views and uuid