Databricks Community

Lakebase Data API

We need to bring the json data from external API and need to load it into lakebase API tables.Please suggest in connecting this.

Databricks best practices for ad-hoc analytics alongside CI/CD

Hi everyone,We currently use CI/CD (Databricks Asset Bundles + Git) for deploying notebooks, jobs and pipelines across DEV and PROD.However, we also have frequent ad-hoc requests from business teams (customer lists, segmentation, one-time analysis), ...

How to Connect Databricks to Snowflake & Query Live Data?

Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...

Data Classification on Foreign Catalog

Can Data Classification be enabled for a foreign catalog?  I haven't found any documentation that says if we are able to or not. Thank you,Mark

Serverless declarative pipeline ingestion from ADLS is running x4 SLOWER randomly

 Hi all,A full refresh of one Auto Loader streaming table in a serverless #Lakeflow (Declarative) pipeline has become about 4x slower (all to do with a single streaming table), and I'd like help understanding why. Nothing on our side changed. The onl...

Chlorophyll

Issue with CDC ingestion pipeline

Hello everyoneI'm trying to setup an ingestion pipeline from a mysql server and keep running into an issue. The setup was validated and the pipeline created. When I run the pipeline the destination tables are created in the schema, but then the follo...

Announcement | The new Genie: autonomous AI for complex enterprise work

Genie is evolving into an autonomous AI coworker that can take on long-running enterprise tasks, not just answer one-off questions. It can monitor progress, follow up when information is missing, and act as new information arrives, within the guardra...

Issues making requests to Databricks from AWS Lambda

Hello,I am getting acquainted with Databricks, of which I have signed up for a free account. I have a Lambda function using Python 3.14 and the databricks-sql-connector library that runs on a schedule, to take data from a file in S3 and put the diffe...

Private Connectivity from Databricks Serverless on AWS to Your On-Prem Data Center

How to connect Databricks serverless workloads to on-premises databases and services over a private network path, using Network Connectivity Configurations and AWS PrivateLink. Why private connectivity to on-prem matters With serverless compute, you ...

CUSTOMER STORY | Mural answers data questions in minutes with Genie Agents

“With Genie, people are asking a lot more questions. They're poking around, experimenting and finding insights no one thought to look for before.” - Wendy Lam, Director of Data Engineering, Mural Mural gives companies like IBM, SAP and Deloitte a sha...

From Kyriba to Lakehouse: A Near-Real-Time Treasury Data Pipeline on Databricks

The two-speed problem of treasury data Corporate treasury asks the same question at two speeds. Where is our money right now? And where was it at the close of business? For many treasury teams the answer lives in Kyriba. Cash flows, balances and FX e...

How to Query Unity Catalog Views from External Engines

Summary External engines can now query Unity Catalog views. Views over Delta tables are served through the Unity REST API (Generally Available). Views over Iceberg tables are served through the Iceberg REST Catalog (Beta).The external engine fetches ...

Request for Guidance: Enabling PostgreSQL CDC in Azure Databricks Lakeflow Connect

Hello Databricks Community,I am using Azure Databricks Lakeflow Connect with a PostgreSQL source.The CDC option is disabled in the PostgreSQL ingestion wizard, and I see the following message:This mode is not available in this workspace for the selec...