Databricks

JKR · 01-12-2024

I have 2 tasks in databricks job workflow first task is of type SQL and SQL task is query.In that query I've declared 2 variables and SET the values by running query.e.g:DECLARE VARIABLE max_timestamp TIMESTAMP DEFAULT '1970-01-01'; SET VARIABLE max_...

JKR · 09-01-2023

Got below failure on scheduled job on interactive cluster and the next scheduled run executed fine.I want to know why this error occurred and how can I prevent it to happen again.And how to debug these errors in future ? com.databricks.backend.commo...

JKR · 09-01-2023

Job is scheduled on interactive cluster, and it failed with below error and in the next scheduled run it ran fine. I want to why this error occurred and how can I prevent from occurring this again.How to debug these types of error? com.databricks.b...

JKR · 06-23-2023

Dear Databricks Certification Team,Unfortunately, I was unable to take the exam as scheduled due to an unforeseen power breakdown in my area. The power outage occurred just before the exam, rendering me unable to access the necessary resources to com...

JKR · 04-25-2023

Getting below error Context: Using Databricks shared interactive cluster for scheduled run multiple parallel jobs at the same time after every 5 mins. When I check Ganglia, driver node's memory reaches almost max and then restart of driver happens an...

JKR · 11-22-2023

@Tharun-Kumar My job was scheduled for every 5 mins and the next scheduled job run executed fine and even job runs after this error one mentioned executed fine without adding these configs, so how does adding the above configurations will help me to ...

JKR · 09-07-2023

@Tharun-Kumar Thanks for sharing this configuration, mentioned error was occurred one time only and the next run executed successfully. Do I still need to add these configurations ? Can you please explain or share any doc to let me know the purpose...

JKR · 08-24-2023

@Kaniz Thank for sharing this, it is really helpful, but my question remains the same that how can we know Ingestion Time Clustering is enabled? As per doc it is enabled by default with DBR 11.2 & above.- Does Ingestion Time Clustering and Liquid cl...

JKR · 08-24-2023

Hi @jose_gonzalez My problem is similar to this.I have 5 interactive clusters running and 24 jobs running on each cluster after every 5 minutes, cannot use job clusters here due to start-up-time and it would miss 5 min SLA. Jobs are running fine but ...

JKR · 08-23-2023

Hi @Kaniz refereing this "Remember that this will only work if you have set up ingestion time clustering for your table". Can you please elobrate how can we setup "ingestion time clustering" for existing non-partitioned tables?

Databricks

User Stats

User Activity

Databricks sql variables and if/else workflow

Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error

Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error

Missed Associate Developer for Apache Spark 3.0 - Python Due to Power outage

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

Re: Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error

Re: Got Failure: com.databricks.backend.common.rpc.SparkDriverExceptions$ReplFatalException error

Re: inegstion time clustering

Re: Cluster Performance

Re: inegstion time clustering