Databricks Community

RESTORE deletes part of the delta table's history

Having a delta table with the history of 15 versions (see screenshot). After running the command:RESTORE TABLE hive_metastore.my_schema.my_table TO VERSION AS OF 6;And then running DESCRIBE HISTORY (see screenshot) it seems that a new version (RESTOR...

Load data from Aurora to Databricks directly

Hi,Does anyone know how to link Aurora to Databricks directly and load data into Databricks automatically on a schedule without any third-party tools in the middle?

Serverless compute vs Job cluster

Hi Guys,For running the job with varying workload what should I use ? Serverless cluster or Job compute ?What are positives and negatives?(I'll be running my notebook from Azure data factory)

Databricks Vs Fabric use case

Hi Team,We've noticed that for some use cases, customers are proposing a architecture with A) Fabric in the Gold layer and reporting in Azure Power BI, while using Databricks for the Bronze and Silver layers. However, we can also have the B) Gold lay...

OOM Issue in Streaming with foreachBatch()

I have a stateless streaming application that uses foreachBatch. This function executes between 10-400 times each hour based on custom logic.  The logic within foreachBatch includes: collect() on very small DataFrames (a few megabytes) --> driver mem...

Autoloader infering struct as a string when reading json data

Hi Everyone,Trying to read JSON files with autoloader is failing to infer the schema correctly, every nested or struct column is being inferred as a string.   spark.readStream.format("cloudFiles") .option("cloudFiles.format", "json") .option("cloud...

Databricks vs snowflake use case comparision

Hi Databricks Team,We see Databricks and Snowflake as very close in terms of features. When trying to convince customers about Databricks' products, we would like to know the key comparisons between Databricks and Snowflake by use case.Regards,Phani

Databricks AI (LLM) Functionalities: Data Privacy and Security

Hi Databricks Team,When leveraging Databricks' AI (LLM) functionalities, such as ai_query and ai_assistant, how does Databricks safeguard customer data and ensure privacy, safety, and security?Regards,Phani

Debugger freezes when calling spark.sql with dbx connect

I have just created a simple bundle with databricks, and is using Databricks connect to debug locally. This is my script:from pyspark.sql import SparkSession, DataFrame def get_taxis(spark: SparkSession) -> DataFrame: return spark.read.table("samp...

I have completed my Databricks Learning Festival and yet havent recieved coupon Code

How to handle , Error parsing WKT: Invalid coordinate value '180' found at position

DBR Version- 16.2spark.databricks.geo.st.enabled trueSQL Query I am running:  %sql WITH points ( SELECT st_astext(st_point(30D, 10D)) AS point_geom UNION SELECT st_astext(st_point(10D, 90D)) AS point_geom UNION SELECT st_astext(st_point(4...

How to store SQL query output columns as variables to be used as parameters for API data call in DAT

I have a sql query which provides me with the below output :Select FirstName,LastName,Title From Default.Name Tony Gonzalez Mr Tom Brady Mr Patricia Carroll MissI would like to store FirstName, LastName & title column output...