Databricks Community

I’m building a Databricks App that continuously queries a SQL Warehouse roughly every 30 seconds to retrieve updated data.To avoid the overhead of repeatedly opening new connections, I’m currently caching the Databricks SQL connection using lru_cache...

I’m trying to understand the networking implications of moving some logic to Databricks Serverless / SDP.My current setup is a notebook running as a job on classic compute, and this works because outbound traffic goes through a NAT Gateway, so we can...

 ’ve created a notebook that uses Spark Structured Streaming and runs continuously, so I’ve deployed the corresponding Databricks job using the continuous trigger mode.What I’d like is for this job to start automatically only in certain environments ...

I’ve built an ingestion pipeline in Databricks consisting of two notebooks:The first notebook calls an external API every four minutes to retrieve the latest available data.Each API call returns approximately 109 rows.The API only exposes the most re...

I’m currently developing a Databricks App, and the app’s service principal needs access to Unity Catalog tables. From what I can tell, it doesn’t seem possible to grant Unity Catalog permissions through DABs yet — only through the UI, based on the cu...