Databricks Community

ImAbhishekTomar · 08-28-2024

I’m trying to drop duplicate in a DF where I have 500B records I’m trying to delete based on multiple columns but this process it’s takes 5h, I try lot of things that available on internet but nothing is works for me.my code is like this.df_1=spark....

ImAbhishekTomar · 10-07-2022

I am facing an error when trying to write data on Kafka using spark stream.#Extract source_stream_df= (spark.readStream .format("cosmos.oltp.changeFeed") .option("spark.cosmos.container", PARM_CONTAINER_NAME) .option("spark.cosmos.read.inferSchema.en...

ImAbhishekTomar · 04-21-2022

Hi,Using the below cosmos DB query it is possible to achieve the expected output, but how can I do the same with spark SQL in Databricks.COSMOSDB QUERY : select c.ReportId,c.ReportName,i.price,p as provider from c join i in in_network join p in i.pr...

ImAbhishekTomar · 02-22-2023

Did anyone find solution for this, I’m also using similar clutter and RAU and data ingestion taking lot of time….?

ImAbhishekTomar · 10-07-2022

There is any other way to check the connect between my Databricks cluster and Kafka broker?

ImAbhishekTomar · 10-07-2022

@Sivaprasad - when I’m telnet my broker and port in Databricks then it’s showing connected.Thanks,Abhishek

Databricks Community

User Stats

User Activity

drop duplicate in 500B records

kafkashaded.org.apache.kafka.common.errors.TimeoutException: topic-downstream-data-nonprod not present in metadata after 60000 ms.

Trying to Flatten My Json using CosmosDB Spark connector - Azure Databricks

Re: Need help to insert huge data into cosmos db from azure data lake storage using databricks

Re: kafkashaded.org.apache.kafka.common.errors.TimeoutException: topic-downstream-data-nonprod not present in metadata after 60000 ms.

Re: kafkashaded.org.apache.kafka.common.errors.TimeoutException: topic-downstream-data-nonprod not present in metadata after 60000 ms.