Databricks Community

Pat

How AI-powered development accelerated my data engineering workflow Watch the Complete Development Process YouTube Video: See the entire 30-minute development sessionThis is a screen recording without voice narration showing the complete developmen...

Pat

Hey,I couldn't find this documented anywhere, but I have been deploying Databricks Workflows (now called Jobs, I believe) with Pipelines using DABs. I have a 1 set of configuration, so there is no place for human error here.When I deploy bundle in DE...

Pat · 05-30-2025

Hey,I’m working on pulling data from AWS SQS into Databricks using Spark custom data sources and DLT (see https://docs.databricks.com/aws/en/pyspark/datasources). I started with a batch reader/writer based on this example: https://medium.com/@zcking/...

Pat · 12-16-2022

Hi community,I don't know what is happening TBH. I have a use case where data is written to the location "dbfs:/mnt/...", don't ask me why it's mounted, it's just a side project. I do believe that data is stored in ADLS2.I've been trying to read the ...

Pat · 11-22-2022

How do you automate table creation and management of privilege / grants on securable objects (tables, views, etc.).I had idea to use terraform to manage above, but terraform doesn't work with managed tables - it allows to create table but then there ...

Pat

Hi @smpa01 ,You cannot create external table on the data in volume:see link to documentation: https://docs.databricks.com/gcp/en/volumes

Pat

@xj2193 I am not sure why you are getting this error, maybe if you could share some code to understand it.Also, you could consider to use `create_auto_cdc_from_snapshot_flow` dlt.create_streaming_table("target") dlt.create_auto_cdc_from_snapshot_f...

Pat

Hi @prakashhinduja ,I have something that might inspire you, as I am currently looking into it.DQX:https://databrickslabs.github.io/dqx/docs/demos/https://www.youtube.com/watch?v=e5Qvx_gnxTEthanks,Pat.

Pat

If someone needs this in the future, like I did.The issue is with your JSON structure. The Databricks CLI uses "changes" with "add" instead of "privilege_assignments" with "privileges".{ "changes": [ { "principal": "mailid", "add": ...

Pat

I would check the network connection between cluster and repository. The error shows pip is currently in the retry phase due to a ConnectTimeoutError, indicating network connectivity issues when trying to reach the package repository.

Databricks Community

User Stats

User Activity

Building DLT Pipelines with Databricks Free Edition and Amazon Q Developer

Pipeline Tags in development mode

Spark custom data sources - SQS streaming reader [DLT]

Reading data from "dbfs:/mnt/"

Automating DDLs and Privilege management

Re: External Table from volume

Re: Delta Live Table - Flow detected an update or delete to one or more rows in the source table

Re: Prakash Hinduja Switzerland (Swiss) How do I implement data quality checks in Databricks workflo

Re: Databricks grant update calatog catlog_name --json @privileges.json not updating privileges

Re: Library installation failed for library due to user error for pypi