Databricks Community

Ramana

Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.When we try to execute the existing jobs with Serverless Compute, if the ...

Ramana

Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.We have several scenarios where we need to send an inline email via Pytho...

Ramana

Hello Community,We have been trying to migrate our jobs from Classic Compute to Serverless Compute. As part of this process, we face several challenges, and this is one of them.When we read CSV or JSON files with multiLine=true, the load becomes sing...

Ramana · 06-05-2025

Last year, we had books from @BrookeWenig, @dennyglee, @JulesDamji, @Holden, @adipolak, and other great authors.Are there any interesting Book Signing Alerts this year at Summit 2025, like we had last year?@Data_AI_summit @Summit2025 @BookSigning#Gen...

Ramana · 06-05-2025

@Hubert-Dudek, I see your post regarding Key Value Secret handling via UI for Databricks on Azure.Is this feature available for Databricks on AWS as well?#Secrets #Scopes #DatabricksOnAWSThanksRamana

Ramana

@K_Anudeep FYI: Serverless Compute doesn't support spark.sql.planChangeLog.level.If we try to set up, the job will fail with [CONFIG_NOT_AVAILABLE] Configuration spark.sql.planChangeLog.level is not available. SQLSTATE: 42K0I error.Classic Compute su...

Ramana

Thank you.But the same job with a limit clause works great. I don't think it is related to the logical plan, but I will look into your suggestion for tracing the issue down. If serverless doesn't work for these basic transformations, it will be tough...

Ramana

Thanks for sharing your thoughts.The job(s) actually failing are simple SELECT * FROM source_table WHERE where_clause with some level of JDBC partitioning and then simple transformations like casting the data types, applying some regex, etc., (may be...

Ramana

@szymon_dybczak it is neither a code issue nor an SMTP host issue, but it is purely related to Serverless Compute.The same code and hostname work perfectly fine on Classic Compute but not on Serverless.I don't think Databricks Serverless can recogniz...

Ramana

@szymon_dybczak I have a list of issues not to use serverless, and this is one of them. Currently, most of my jobs use Classic compute.If I find or hear something from the Serverless team, I will let you know.

Databricks Community

User Stats

User Activity

Serverless Compute - Spark - Jobs failing with Max iterations (1000) reached for batch Resolution

Serverless Compute - Python - Custom Emails via SMTP (smtplib.SMTP(host_name)) - Any alternative?

Serverless Compute - pySpark - Any alternative for rdd.getNumPartitions()

Any interesting Book Signing Alerts this year at Summit 2025 like we had last year?

Databricks Key Vault Secret - Is it available in Databricks on AWS?

Re: Serverless Compute - Spark - Jobs failing with Max iterations (1000) reached for batch Resolutio

Re: Serverless Compute - Spark - Jobs failing with Max iterations (1000) reached for batch Resolutio

Re: Serverless Compute - Spark - Jobs failing with Max iterations (1000) reached for batch Resolutio

Re: Serverless Compute - Python - Custom Emails via SMTP (smtplib.SMTP(host_name)) - Any alternative

Re: Serverless Compute - pySpark - Any alternative for rdd.getNumPartitions()