Databricks Community

I guess I was a bit over enthusiastic by accepting the answer.When I run the following on the single object array of arrays (as shown in the original post) I get a single row with column "value" and value null. from pyspark.sql import functions as F,...

For now we are trying to contact Databricks, In worst case scenario we were planning to shade the dependencies we need.would love to hear what has worked for you.Best,Shimon

Your Databricks question about workers versus executors. Many teams encounter the same sizing and configuration issues when evaluating a migration. At Kanerika, we help companies plan cluster architecture, optimize Spark workloads, and avoid overspen...

Possible reasons:1. Setting spot_bid_max_price = -1 is not accepted by Azure poolsAzure Databricks only accepts:0 → on-demand onlypositive numbers → max spot price-1 is allowed in cluster policies, but not inside pools, so validation never completes....

Hi @SRJDB ,Could you execute my_dataframe.printSchema() and attach result here? 

@ManojkMohan as mentioned in the first post I already used dict(spark.conf.getAll()).get(key, default) where possible.However the problem stands when importing modules, like:- from dbacademy import dbgems- from dbacademy.dbhelper import DBAcademyHelp...

private link from serverless, as probably you are not allowing public internet access. Configure private connectivity to Azure resources - Azure Databricks | Microsoft Learn you need to add both dfs and blob

Hi there, I vibe-coded* a query where I was able to derive most of your events from the system tables: system.lakeflow.jobssystem.lakeflow.job_run_timelinesystem.lakeflow.job_task_run_timeline If you have SELECT access to system tables, this could b...

Really cool project! The mood-based movie matching and conversational memory make the whole discovery experience feel way more intuitive. It’s interesting because most people still browse platforms manually — like on streaming App — but your system s...

@Naveenkumar1811 Since your silver is a streaming job, there can be lots of files and metadata being created based on your write interval and frequency of new data. If there are more files being created in few mins, it potentially leads to small file...

Hello,IMHO, having a HTTP related task in a Spark cluster is an anti-pattern. This kind of code executes at the Driver, it will be synchronous and adds overhead. This is one of the reasons, DLT (or SDP - Spark Declarative Pipeline) does not have REST...