Databricks Community

data_pulse

Databricks mentions in the DBR 19 release notes that Auto CDF removes write-time overhead and can make MERGE and UPDATE operations about 15% faster on tables queried for changesFor anyone who has moved an existing pipeline from legacy CDF to Auto CDF...

data_pulse

Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...

data_pulse

I am looking at a scenario for team-level cost attribution using system.billing.usage and have a tagging gap for Databricks-managed services such as Predictive Optimization and Data Quality Monitoring.Tried adding a team/ownership tag directly to the...

data_pulse

@qduan Also do check the backend state before re-creating the assistant. I had tested a similar failure path and saw the assistant remain in CREATING while the file source stayed in UPDATING, with no error_info surfaced yet.Few useful checks:import r...

data_pulse

@SKiruthiga I’m using Snowflake through a Unity Catalog federated connection with PEM private-key authentication. That keeps credentials out of the Spark code and lets Snowflake exposed through a foreign catalog.It definitely is not the same as Snowf...

data_pulse

@Danish11052000 A feasible workaround is to use the Databricks Account Workspaces API to fetch the GCP project_id alongside workspace_id. The workspace API schema exposes it under cloud_resource_container.gcp.project_id. Referenceurl = f"https://acco...

data_pulse

@LiresaFerizaj The other replies already cover the retention/performance and full refresh questions well, I only did a quick validation around point 1 : late events after the tombstone retention window.I used SCD2 AUTO CDC with: "pipelines.cdc.tombst...

data_pulse

@koosha I tested this with a small Genie Agent example using a table with:ticket_id - display value the user should click, for example INC001.incident_url - Actual URL for that ticket, like:https://example.com/incidents/INC001Returning those as separ...

Databricks Community

User Stats

User Activity

Anyone migrated from legacy CDF to Auto CDF yet?

Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

Re: Knowledge Assistant fails to load after indexing; generated AI Search index cannot be reused

Re: Connecting to Snowflake via Workload Identity Federation

Re: How to bring the project id info into the databricks billing usage table on GCP

Re: AUTO CDC SCD Type 2 with late-arriving deletes: edge cases the docs don't cover

Re: Genie agent hyperlink creation