Databricks Community

I'm working with Databricks declarative pipelines and have defined a custom PySpark data source (CDS) in its own standalone Python module. I include this module as part of the pipeline resources. What I find interesting is that, even without explicit...

I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache...

I’m trying to understand the networking implications of moving some logic to Databricks Serverless / SDP.My current setup is a notebook running as a job on classic compute, and this works because outbound traffic goes through a NAT Gateway, so we can...

 ’ve created a notebook that uses Spark Structured Streaming and runs continuously, so I’ve deployed the corresponding Databricks job using the continuous trigger mode.What I’d like is for this job to start automatically only in certain environments ...

I’ve built an ingestion pipeline in Databricks consisting of two notebooks:The first notebook calls an external API every four minutes to retrieve the latest available data.Each API call returns approximately 109 rows.The API only exposes the most re...