- 500 Views
- 4 replies
- 0 kudos
AI/BI Dashboards
Hi AllMy client is looking to create a few dashboards in DataBricks instead of Power BI. We already have a template in Power BI (header for each page, information about out BI Page) in each of our power bi report which has colors, fonts, images etc.H...
- 500 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @km1837, Databricks AI/BI Dashboards have solid theming and organisational capabilities that should cover most of what you are used to in Power BI. AI/BI Dashboards don't have a "template file" concept the way Power BI does with .pbit files, but y...
- 0 kudos
- 735 Views
- 4 replies
- 1 kudos
Resolved! Auto Loader stream fails on RocksDB checkpoint after enabling managed file events
Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...
- 735 Views
- 4 replies
- 1 kudos
- 1 kudos
After further investigation into it, have understood the root cause and the fix for it.After enabling Auto Loader managed file events across the platform: .option("cloudFiles.useManagedFileEvents", "true")the failed stream started discovering RocksDB...
- 1 kudos
- 747 Views
- 3 replies
- 2 kudos
Resolved! Databricks Apps - Deployment process - Need help !!
Hi all,Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime...
- 747 Views
- 3 replies
- 2 kudos
- 2 kudos
Thank you for the details. Yes, I agree we should adopt the DABs for the deployment and define the resources while creating the app ( which can be consider as as solution). However if the permission module is managed separately, lets say via terrafor...
- 2 kudos
- 798 Views
- 2 replies
- 3 kudos
Resolved! Best Practice for Handling Schema Evolution with Auto Loader in Production?
Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...
- 798 Views
- 2 replies
- 3 kudos
- 3 kudos
You can absolutely leverage Databricks Genie and the new Genie Ontology to handle downstream changes caused by addNewColumnsWithTypeWidening, and handle the changes using Genie ontology, but it requires understanding the division of labor between you...
- 3 kudos
- 397 Views
- 1 replies
- 0 kudos
OAuth M2M (client-credentials) - getting error with github run
I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token...
- 397 Views
- 1 replies
- 0 kudos
- 0 kudos
further refined this problem with below if anyone has any idea how I could address? thanks. Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so what...
- 0 kudos
- 322 Views
- 2 replies
- 0 kudos
How to create monotonic function to incrementally add obj_id for datasource in pyspark
I have used monotonically_increasing_id() to add unique to datasource .However , this does not add unique id in increasing number ..Its kind of random.How to perform this in pyspark.
- 322 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey So monotonically_increasing_id() only promises the numbers go up and are unique, not that they're neat like 1, 2, 3. Under the hood it bakes the partition number into the ID, so each partition jumps ahead by billions.If you let me know what you'...
- 0 kudos
- 586 Views
- 2 replies
- 4 kudos
Resolved! Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes
Hi community,I’m currently architecting a multi-workspace environment under Unity Catalog, and I’m looking for the most sustainable way to manage Service Principal access to raw data landing zones.We are debating between using External Locations (sco...
- 586 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi,On question one, I would reframe the choice rather than answer it directly. Managed volumes sit in Databricks controlled storage, which means an external producer cannot write into them through Unity Catalog. If your landing zone is fed by systems...
- 4 kudos
- 630 Views
- 2 replies
- 3 kudos
Resolved! Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines
Hi everyone,As we look to modernize our Delta tables, I’m evaluating the move from Z-Ordering to Liquid Clustering, particularly for our historical datasets that exceed 500TB.While the benefits of Liquid Clustering (avoiding over-partitioning and bet...
- 630 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,The good news is that the migration is much cheaper than you are assuming, because the full rewrite is optional.When you run ALTER TABLE with CLUSTER BY, existing data is not rewritten. Subsequent writes and normal OPTIMIZE runs use the new cluste...
- 3 kudos
- 606 Views
- 2 replies
- 2 kudos
Resolved! Oracle NUMBER → DecimalType(38,10) on ingestion, and silver layer best practices
Hello everyone,I've just started a new position and my goal is to build a Data Lakehouse with Databricks. I have no previous experience with Databricks and I'm being helped by an external company. The goal is to centralize all our data in Databricks....
- 606 Views
- 2 replies
- 2 kudos
- 2 kudos
@faruk Handling Oracle NUMBER in the ingestion pipelineThe Oracle JDBC driver maps an NUMBER (no precision, no scale) to DecimalType(38,10) as it must pick a fixed scale for a type system that has no equivalent to Oracle's unconstrained decimal. You ...
- 2 kudos
- 839 Views
- 8 replies
- 2 kudos
shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log
SetupSingleton Python logger (logging.FileHandler) writing .log files during ADF-orchestrated notebook runs (one notebook per file). Files need to land in a UC Volume.Context on the migration path — this workflow has been reshaped repeatedly by the m...
- 839 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @DoTA , I have been using a singleton logger as there are helper classes being used within the notebook which will also need to use the logger.
- 2 kudos
- 452 Views
- 3 replies
- 0 kudos
Handling New Columns in a Databricks Data Pipeline
Hi everyone,I have a question about handling schema changes during data ingestion in Databricks.Let’s say an e-commerce company receives customer data every day with the following columns:Customer_id, name, cityAfter a few months, the source system a...
- 452 Views
- 3 replies
- 0 kudos
- 0 kudos
@gowri_databrick Auto Loader automatically detects and handles new columns when ingesting files. When your source adds the email column, Auto Loader's checkpoint tracks the schema change and can either rescue unknown columns into _rescued_data (safe ...
- 0 kudos
- 265 Views
- 1 replies
- 0 kudos
Databricks Secrets
Recently Databricks Secrets became generally available.Now, you can manage your secrets whitin the catalog using the namespace: catalog.schema.secretHowever, you can do the opposite of what you used to do: Connect Unity Catalog secrets to Azure Key V...
- 265 Views
- 1 replies
- 0 kudos
- 0 kudos
One dimension I would add, since it usually decides it for regulated shops: governance and audit. - UC-managed secrets (catalog.schema.secret) inherit UC's permission model and land in the same audit / system tables as everything else, so "who read t...
- 0 kudos
- 477 Views
- 2 replies
- 2 kudos
Resolved! Auto Loader Strategy: Balancing Cloud Notification Costs vs. Directory Listing in Massive Migrations
Hi everyone,I’m currently architecting a high-volume migration for a platform that generates millions of small files daily across thousands of prefixes. As I evaluate the best ingestion strategy using Databricks Auto Loader, I’m running into an archi...
- 477 Views
- 2 replies
- 2 kudos
- 2 kudos
In the context of enterprise-grade cloud migrations (think legacy databases or petabyte-scale data lakes ingestion into Databricks Delta Lake), the industry standard approach involves splitting the ingestion stack into 2:History (Initial Bulk Load) a...
- 2 kudos
- 687 Views
- 2 replies
- 2 kudos
Resolved! Managing Schema Drift and Evolution in Spark Declarative Pipelines (SDP)
Hi everyone,I’m currently diving into the Spark Declarative Pipelines (SDP) framework and evaluating it for our upcoming migration of a high-velocity event-driven pipeline.One area I’m trying to solidify is our strategy for Schema Evolution. In tradi...
- 687 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi,One framing correction that might save you time: there is no separate schema drift control at the declarative layer. SDP handles orchestration and dependencies between flows. Schema evolution is still owned by the source reader and by the target t...
- 2 kudos
- 778 Views
- 5 replies
- 4 kudos
Resolved! Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?
I am looking at a scenario for team-level cost attribution using system.billing.usage and have a tagging gap for Databricks-managed services such as Predictive Optimization and Data Quality Monitoring.Tried adding a team/ownership tag directly to the...
- 778 Views
- 5 replies
- 4 kudos
- 4 kudos
@stbjelcevic Thanks, that was helpful and gave another way to solve this use case.Instead of maintaining a separate table for object to team mapping table, can now use UC tags as the ownership source with precedence:table tag -> schema tag -> catalog...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 585 | |
| 349 | |
| 287 |
