- 945 Views
- 2 replies
- 3 kudos
Resolved! Best Practice for Handling Schema Evolution with Auto Loader in Production?
Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...
- 945 Views
- 2 replies
- 3 kudos
- 3 kudos
You can absolutely leverage Databricks Genie and the new Genie Ontology to handle downstream changes caused by addNewColumnsWithTypeWidening, and handle the changes using Genie ontology, but it requires understanding the division of labor between you...
- 3 kudos
- 469 Views
- 1 replies
- 0 kudos
OAuth M2M (client-credentials) - getting error with github run
I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token...
- 469 Views
- 1 replies
- 0 kudos
- 0 kudos
further refined this problem with below if anyone has any idea how I could address? thanks. Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so what...
- 0 kudos
- 376 Views
- 2 replies
- 0 kudos
How to create monotonic function to incrementally add obj_id for datasource in pyspark
I have used monotonically_increasing_id() to add unique to datasource .However , this does not add unique id in increasing number ..Its kind of random.How to perform this in pyspark.
- 376 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey So monotonically_increasing_id() only promises the numbers go up and are unique, not that they're neat like 1, 2, 3. Under the hood it bakes the partition number into the ID, so each partition jumps ahead by billions.If you let me know what you'...
- 0 kudos
- 660 Views
- 2 replies
- 4 kudos
Resolved! Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes
Hi community,I’m currently architecting a multi-workspace environment under Unity Catalog, and I’m looking for the most sustainable way to manage Service Principal access to raw data landing zones.We are debating between using External Locations (sco...
- 660 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi,On question one, I would reframe the choice rather than answer it directly. Managed volumes sit in Databricks controlled storage, which means an external producer cannot write into them through Unity Catalog. If your landing zone is fed by systems...
- 4 kudos
- 697 Views
- 2 replies
- 3 kudos
Resolved! Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines
Hi everyone,As we look to modernize our Delta tables, I’m evaluating the move from Z-Ordering to Liquid Clustering, particularly for our historical datasets that exceed 500TB.While the benefits of Liquid Clustering (avoiding over-partitioning and bet...
- 697 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,The good news is that the migration is much cheaper than you are assuming, because the full rewrite is optional.When you run ALTER TABLE with CLUSTER BY, existing data is not rewritten. Subsequent writes and normal OPTIMIZE runs use the new cluste...
- 3 kudos
- 707 Views
- 2 replies
- 2 kudos
Resolved! Oracle NUMBER → DecimalType(38,10) on ingestion, and silver layer best practices
Hello everyone,I've just started a new position and my goal is to build a Data Lakehouse with Databricks. I have no previous experience with Databricks and I'm being helped by an external company. The goal is to centralize all our data in Databricks....
- 707 Views
- 2 replies
- 2 kudos
- 2 kudos
@faruk Handling Oracle NUMBER in the ingestion pipelineThe Oracle JDBC driver maps an NUMBER (no precision, no scale) to DecimalType(38,10) as it must pick a fixed scale for a type system that has no equivalent to Oracle's unconstrained decimal. You ...
- 2 kudos
- 1007 Views
- 8 replies
- 2 kudos
Resolved! shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log
SetupSingleton Python logger (logging.FileHandler) writing .log files during ADF-orchestrated notebook runs (one notebook per file). Files need to land in a UC Volume.Context on the migration path — this workflow has been reshaped repeatedly by the m...
- 1007 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @DoTA , I have been using a singleton logger as there are helper classes being used within the notebook which will also need to use the logger.
- 2 kudos
- 539 Views
- 3 replies
- 0 kudos
Handling New Columns in a Databricks Data Pipeline
Hi everyone,I have a question about handling schema changes during data ingestion in Databricks.Let’s say an e-commerce company receives customer data every day with the following columns:Customer_id, name, cityAfter a few months, the source system a...
- 539 Views
- 3 replies
- 0 kudos
- 0 kudos
@gowri_databrick Auto Loader automatically detects and handles new columns when ingesting files. When your source adds the email column, Auto Loader's checkpoint tracks the schema change and can either rescue unknown columns into _rescued_data (safe ...
- 0 kudos
- 305 Views
- 1 replies
- 0 kudos
Databricks Secrets
Recently Databricks Secrets became generally available.Now, you can manage your secrets whitin the catalog using the namespace: catalog.schema.secretHowever, you can do the opposite of what you used to do: Connect Unity Catalog secrets to Azure Key V...
- 305 Views
- 1 replies
- 0 kudos
- 0 kudos
One dimension I would add, since it usually decides it for regulated shops: governance and audit. - UC-managed secrets (catalog.schema.secret) inherit UC's permission model and land in the same audit / system tables as everything else, so "who read t...
- 0 kudos
- 513 Views
- 2 replies
- 2 kudos
Resolved! Auto Loader Strategy: Balancing Cloud Notification Costs vs. Directory Listing in Massive Migrations
Hi everyone,I’m currently architecting a high-volume migration for a platform that generates millions of small files daily across thousands of prefixes. As I evaluate the best ingestion strategy using Databricks Auto Loader, I’m running into an archi...
- 513 Views
- 2 replies
- 2 kudos
- 2 kudos
In the context of enterprise-grade cloud migrations (think legacy databases or petabyte-scale data lakes ingestion into Databricks Delta Lake), the industry standard approach involves splitting the ingestion stack into 2:History (Initial Bulk Load) a...
- 2 kudos
- 835 Views
- 2 replies
- 2 kudos
Resolved! Managing Schema Drift and Evolution in Spark Declarative Pipelines (SDP)
Hi everyone,I’m currently diving into the Spark Declarative Pipelines (SDP) framework and evaluating it for our upcoming migration of a high-velocity event-driven pipeline.One area I’m trying to solidify is our strategy for Schema Evolution. In tradi...
- 835 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,One framing correction that might save you time: there is no separate schema drift control at the declarative layer. SDP handles orchestration and dependencies between flows. Schema evolution is still owned by the source reader and by the target t...
- 2 kudos
- 874 Views
- 5 replies
- 4 kudos
Resolved! Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?
I am looking at a scenario for team-level cost attribution using system.billing.usage and have a tagging gap for Databricks-managed services such as Predictive Optimization and Data Quality Monitoring.Tried adding a team/ownership tag directly to the...
- 874 Views
- 5 replies
- 4 kudos
- 4 kudos
@stbjelcevic Thanks, that was helpful and gave another way to solve this use case.Instead of maintaining a separate table for object to team mapping table, can now use UC tags as the ownership source with precedence:table tag -> schema tag -> catalog...
- 4 kudos
- 1145 Views
- 6 replies
- 4 kudos
Resolved! How are you separating dev, staging and prod in Unity Catalog without duplicating everything?
Hi everyone, We are trying to settle on an environment strategy and keep going back and forth, so I would like to hear how other teams actually landed on this rather than what the reference architecture suggests. Our situation. One Databricks account...
- 1145 Views
- 6 replies
- 4 kudos
- 4 kudos
Here is, in summary, how I am facing that issue in some real projects in Deloitte:Unity Catalog Metastore│├── Workspace DEV│ └── sales_dev│ └── finance_dev│ └── customer_dev│├── Workspace QA│ └── sales_qa│ └── finance_qa│ └── customer_qa│├── Workspac...
- 4 kudos
- 312 Views
- 1 replies
- 1 kudos
BOOTSTRAP_TIMEOUT on cluster start (Southeast Asia)
Our cluster start fails with:BOOTSTRAP_TIMEOUT: Instance bootstrap failed command: Bootstrap_e2e Instance bootstrap inferred timeout reason: UnknownReason Failure message: Bootstrap is terminated spontaneously by VM because self timeout was reached. ...
- 312 Views
- 1 replies
- 1 kudos
- 1 kudos
Widening the NSG service tag to all regions is a good first step but it rarely fixes this on its own. The AzureDatabricks service tag only covers the control plane traffic. What most people miss is that during bootstrap, the VM also needs to reach Az...
- 1 kudos
- 455 Views
- 1 replies
- 3 kudos
Resolved! Now that Unity Catalog manages Iceberg natively, is anyone actually switching?
Hi everyone,Unity Catalog managed Iceberg tables went GA earlier this year, and they arrived with the features that used to be the reason to stay on Delta: liquid clustering, predictive optimization, materialized views, streaming tables. On paper the...
- 455 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Islam_hoti,The Reality on the Ground: We have experimented with Iceberg, but for our primary production workloads, we remain 100% Delta Lake. Here is our internal "Messy Truth" on why:The "Inertia" Factor: You’re right, inertia is a legitimate en...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
