- 66 Views
- 3 replies
- 2 kudos
Schema changes notification
Hi all,Sometimes an upstream source adds a new column without letting us know. We do not want the Databricks workflow to fail when this happens, but we would like to be notified so we can review the new column and decide whether to include it in our...
- 66 Views
- 3 replies
- 2 kudos
- 2 kudos
@pharma70 if this ingestion is using Auto Loader, cloudFiles.schemaEvolutionMode = "rescue" seems like a good fit here. In that mode Auto Loader keeps the stream running, does not evolve the schema, and puts new fields into _rescued_data.You can then...
- 2 kudos
- 132 Views
- 7 replies
- 1 kudos
Resolved! Serverless Capabilities Not Available In My Workspace
Hello,My AWS Databricks account is paid Premium and has serverless compute enabled but the only types I see for SQL warehouses are Pro and Classic.I created two workspaces each with “Use serverless compute with default storage,” and "Use your existin...
- 132 Views
- 7 replies
- 1 kudos
- 1 kudos
The core issue with your prototype is that the file discovery step is being duplicated: Auto Loader already does incremental, stateful file discovery internally (and can use cloud-native file notifications instead of directory listing), but your dbut...
- 1 kudos
- 1427 Views
- 4 replies
- 3 kudos
Resolved! Missing upstream column lineage missing from api call after some time
I ran the following piece of code on 2 occasions.table_name = 'full path of table"lineage = w.api_client.do("GET",f"/api/2.0/lineage-tracking/column-lineage",body={"table_name": table_name,"column_name": "column_x"})u_lineage_df = spark.createDataFra...
- 1427 Views
- 4 replies
- 3 kudos
- 3 kudos
@Mario_D wrote:I ran the following piece of code on 2 occasions.table_name = 'full path of table"lineage = w.api_client.do("GET",f"/api/2.0/lineage-tracking/column-lineage",body={"table_name": table_name,"column_name": "column_x"})u_lineage_df = spar...
- 3 kudos
- 79 Views
- 3 replies
- 0 kudos
Failed to convert to managed table: PrivilegedGenerateTemporaryTableCredential disable
Hello everyone, I noticed that when I tried to convert some external table to managed one using the alter table catalog.schema.table_name set managed command (like suggested by databricks) sometimes I will encounter this error: Error running query: c...
- 79 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Davide,The conversion is designed to be resumable. It copies data and the Delta log to the managed location first, then switches the metadata, and the page says plainly that if the command fails you can retry it and it "resumes from where it left ...
- 0 kudos
- 146 Views
- 2 replies
- 0 kudos
Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale
Hi everyone, we are designing our security layer in Unity Catalog. We are debating between using Row-Level Security (RLS) predicates versus standard SQL Views for masking PII data.When applying complex RLS predicates to tables with >100M rows, have ...
- 146 Views
- 2 replies
- 0 kudos
- 0 kudos
I advise RLS because as Tomaz said: you cannot apply time travel with masks or filters, also clones are not supported and MERGE does not work with them either.
- 0 kudos
- 126 Views
- 1 replies
- 0 kudos
Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption
Hi everyone, we are defining our PII strategy in Unity Catalog. We are split on whether to: A) Redact/Hash PII at the Silver layer (permanent change), or B) Keep PII in Silver and use Dynamic Data Masking at the Gold/View layer.Does "Redact at Inges...
- 126 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Khasim_1, I would keep the PII in Silver within your regulatory and internal retention limits and control access with Unity Catalog's row filters and column masks, rather than redacting permanently. The reason is the one you already gave. A perma...
- 0 kudos
- 140 Views
- 3 replies
- 0 kudos
Data ingestion: Setting up a connector for Postgres with CDC enabled.
Hello,I'm trying to ingest data from my RDS instance and set up change data capture on Databricks. Everything on the Postgres side has been done so that replication is possible, but CDC is still greyed out. I have asked Claude, Gemini, and ChatGPT. I...
- 140 Views
- 3 replies
- 0 kudos
- 0 kudos
Quick question, being that my account is an individual account, how do I go about this "Contact your Databricks account team to request access"
- 0 kudos
- 152 Views
- 2 replies
- 2 kudos
Best practice to enforcing row-level security across multiple catalogs sharing same schema structure
Hi Everyone,We have identical schema structures replicated across three catalogs (dev, qa, prod) to support environment isolation. We now need to apply row-level security (e.g., restricting sales reps to only see their own region's data) consistently...
- 152 Views
- 2 replies
- 2 kudos
- 2 kudos
Agree with @ThomazNeto - only thing I would add is your point 3 requirement: you could also consider using Terraform instead DABs or job for managing the ABAC policies across multiple environments. It does not support the beta features i.e. metastor...
- 2 kudos
- 135 Views
- 1 replies
- 0 kudos
Managing "Secret" Injection in DABs across Dev/Stage/Prod
Hi everyone, I am fully moving our team to Databricks Asset Bundles (DABs) for CI/CD, but I’m struggling with the "Secret" management pattern.How are you handling the injection of secrets (like API keys for ingestion) into DABs without hardcoding any...
- 135 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Khasim,No secret value ever goes into databricks.yml. Bundle variables get resolved at deploy time and rendered into the job definition in the workspace, so anyone who can open the job can read them. Only names and references live in the bundle.Se...
- 0 kudos
- 220 Views
- 3 replies
- 0 kudos
Predictive Optimization for Streaming tables in Lakeflow pipelines
On July 2025 https://www.databricks.com/blog/whats-new-lakeflow-declarative-pipelines-july-2025 predictive optimization was enabled for all UC managed Lakeflow pipelines. I was wondering if there is a possibility now to disable it for Streaming table...
- 220 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Gilk, Before the how, can I ask what is driving the wish to turn it off? For most pipeline tables predictive optimization is doing useful work, running OPTIMIZE, VACUUM, and ANALYZE on serverless compute so file sizes, storage, and statistics sta...
- 0 kudos
- 70 Views
- 0 replies
- 2 kudos
How I explain account vs workspace vs Unity Catalog metastore to new platform engineers
Sharing the explanation that finally stopped the same question coming back every onboarding.The account is the organisation-level container, normally one per cloud provider, and it owns billing, account admins and identities. A workspace is a single ...
- 70 Views
- 0 replies
- 2 kudos
- 258 Views
- 4 replies
- 0 kudos
Lakeflow connect CT pipeline keeps running multiple connections in source
I'm working on an ingestion pipeline. I was adding tables to it gradually so that we can monitor the load on our prod server side by side and with my last set of tables added (with them, all the heavy duty tables were inside that pipeline), the load ...
- 258 Views
- 4 replies
- 0 kudos
- 0 kudos
For a standard connector, the gateway continuously extracts snapshots, change logs, and metadata, while a separately scheduled serverless pipeline applies staged data (connector components). Changing the serverless pipeline schedule therefore does no...
- 0 kudos
- 188 Views
- 2 replies
- 0 kudos
Databricks Free Edition
Hi,I just signed up for Databricks free edition to start my learning journey in Data Engineering on it.Want to know for how long it remains free?Thank you.
- 188 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @kabiromohd, Welcome to Databricks, and a great choice for starting your Data Engineering journey! The short answer is that Databricks Free Edition is forever free. There is no trial period or expiration date. You can use it indefinitely as lo...
- 0 kudos
- 90 Views
- 0 replies
- 0 kudos
Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle
Hi everyone,Hi everyone, I’m architecting a data-sharing program using Delta Sharing for external partners who do not have a Databricks workspace.How are others automating the recipient credential lifecycle? Do you have an automated process for rotat...
- 90 Views
- 0 replies
- 0 kudos
- 1017 Views
- 9 replies
- 2 kudos
Resolved! Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 1017 Views
- 9 replies
- 2 kudos
- 2 kudos
A single Auto Loader stream with filename-based routing looks much cleaner here, especially since new object types can appear without updating the pipeline.
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
