- 838 Views
- 6 replies
- 4 kudos
Resolved! How are you separating dev, staging and prod in Unity Catalog without duplicating everything?
Hi everyone, We are trying to settle on an environment strategy and keep going back and forth, so I would like to hear how other teams actually landed on this rather than what the reference architecture suggests. Our situation. One Databricks account...
- 838 Views
- 6 replies
- 4 kudos
- 4 kudos
Here is, in summary, how I am facing that issue in some real projects in Deloitte:Unity Catalog Metastore│├── Workspace DEV│ └── sales_dev│ └── finance_dev│ └── customer_dev│├── Workspace QA│ └── sales_qa│ └── finance_qa│ └── customer_qa│├── Workspac...
- 4 kudos
- 238 Views
- 1 replies
- 1 kudos
BOOTSTRAP_TIMEOUT on cluster start (Southeast Asia)
Our cluster start fails with:BOOTSTRAP_TIMEOUT: Instance bootstrap failed command: Bootstrap_e2e Instance bootstrap inferred timeout reason: UnknownReason Failure message: Bootstrap is terminated spontaneously by VM because self timeout was reached. ...
- 238 Views
- 1 replies
- 1 kudos
- 1 kudos
Widening the NSG service tag to all regions is a good first step but it rarely fixes this on its own. The AzureDatabricks service tag only covers the control plane traffic. What most people miss is that during bootstrap, the VM also needs to reach Az...
- 1 kudos
- 374 Views
- 1 replies
- 3 kudos
Resolved! Now that Unity Catalog manages Iceberg natively, is anyone actually switching?
Hi everyone,Unity Catalog managed Iceberg tables went GA earlier this year, and they arrived with the features that used to be the reason to stay on Delta: liquid clustering, predictive optimization, materialized views, streaming tables. On paper the...
- 374 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Islam_hoti,The Reality on the Ground: We have experimented with Iceberg, but for our primary production workloads, we remain 100% Delta Lake. Here is our internal "Messy Truth" on why:The "Inertia" Factor: You’re right, inertia is a legitimate en...
- 3 kudos
- 409 Views
- 4 replies
- 0 kudos
Multiple gateway pipeline for same database
Hi Team,To do the POC on Lakeflow Connect Ingestion Pipeline, we directly connected the sql server prod DB and build the pipeline it is running successfully. Now, My question is we need to move that to UAT and Prod. In Dev we have created througn UI ...
- 409 Views
- 4 replies
- 0 kudos
- 0 kudos
@srikanthp24 , To your core question: multiple gateway pipelines against the same source database don't interfere with each other. Each gateway keeps its own independent state, its own staging volume, and its own CDC/change-tracking cursor, so a sepa...
- 0 kudos
- 268 Views
- 1 replies
- 1 kudos
Databricks Dashboard Embedded Ask Genie
Hi ,I have created Dashboard on Databricks using sql datasets and published with " Shared Data Permission " with the business user . he can see the data and charts but unable to use "Ask Genie " functionality . Do I need to grant Use Catalog , schema...
- 268 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @pragya17 ,You are correct that Shared data permission should not require the business user to have direct USE CATALOG, USE SCHEMA, or SELECT permissions on the underlying data. With embedded/shared credentials, the autogenerated Genie Agent shoul...
- 1 kudos
- 416 Views
- 2 replies
- 0 kudos
Resolved! Can't access abfss data in azure databricks when providing shared key (fighting UC?)
The "no isolation shared" clusters are going to be killed in the next month. This came as a surprise to me. I should have been paying closer attention. We don't heavily use UC since our data is published to the business via Fabric. All the spark ...
- 416 Views
- 2 replies
- 0 kudos
- 0 kudos
I found another reply where the customer was required to add the service principal's full GRANT to the external locations. Seems very odd, and almost less secure than what I was doing with the shared access key.I really think there needs to be a ne...
- 0 kudos
- 12291 Views
- 2 replies
- 3 kudos
Ingesting complex/unstructured data
Hi there,my company is reasonably new to using Databricks, and we're running our first PoCs. Some of the data we have structured/reasonably structured, so it drops into a bucket, we point a notebook at it, and all is well and DeltaThe problem is ari...
- 12291 Views
- 2 replies
- 3 kudos
- 3 kudos
Ingesting complex or unstructured data works best when you break the process into stages rather than trying to analyze everything at once. First, collect the raw data, then clean it, remove duplicates, standardize formats, and extract only the fields...
- 3 kudos
- 614 Views
- 5 replies
- 4 kudos
Resolved! How are you attributing serverless costs back to individual jobs and teams?
Hi everyone,We moved most of our workloads to serverless over the last few months and the performance side has been fine. The part we did not plan for is cost attribution.On classic compute this was easy. One cluster per team, tags on the cluster, an...
- 614 Views
- 5 replies
- 4 kudos
- 4 kudos
@Islam_hoti the most defensible model is to separate measured attribution from allocation policy instead of forcing every DBU into a team bucket.Exact workload attribution. Use system.billing.usage as the ledger. For serverless jobs, group by workspa...
- 4 kudos
- 478 Views
- 5 replies
- 1 kudos
Serverless compute resolves some public domains but not others
Hi everyone,I’m troubleshooting outbound internet access from Databricks serverless compute and am seeing inconsistent DNS resolution.For example, the GitHub API domain resolves successfully:api [dot] github [dot] comDNS: PASS -> DNS resolved success...
- 478 Views
- 5 replies
- 1 kudos
- 1 kudos
@LakehouseLad I also tested the same from a Premium tier workspace, and the same public domains resolve successfully there. So this does not appear to be a general Premium tier limitation.In the screenshot you posted is showing the DNS request as DRO...
- 1 kudos
- 871 Views
- 8 replies
- 4 kudos
Understanding Parquet File Storage for Large Datasets
Hi everyone,I’m learning about Parquet files and how they are used in Databricks for storing large datasets.I’m trying to understand how column-based storage works in a practical situation.For example, suppose an e-commerce company has 500 million or...
- 871 Views
- 8 replies
- 4 kudos
- 4 kudos
In addition to my previous picture, I'd add the following explanation. In Databricks, Partition Pruning, Data Skipping and Parquet’s columnar format work together to minimize I/O.1. Partition Pruning → eliminates partitions.If the table is partitione...
- 4 kudos
- 417 Views
- 3 replies
- 0 kudos
What is a Data Skipping in Delta Lake?
Hi everyone,I’m learning about Delta Lake performance and came across data skipping.I understand that it can help Databricks avoid reading unnecessary data when running queries, but I’d like to understand its purpose more clearly.For example, if an o...
- 417 Views
- 3 replies
- 0 kudos
- 0 kudos
I think the below picture will help a bit. Take a look at section 2 for "data skipping":
- 0 kudos
- 603 Views
- 5 replies
- 3 kudos
Partitioning vs Liquid Clustering (per-table):
Can PARTITION BY and CLUSTER BY (Liquid Clustering) be used simultaneously on the same table? If we use only PARTITION BY, is there a negative performance impact on materialized-view refreshes in Silver/Gold? Since materialized views read only increm...
- 603 Views
- 5 replies
- 3 kudos
- 3 kudos
Not possible at all BUT here is a summary comparing both methods:I hope it helps.
- 3 kudos
- 426 Views
- 4 replies
- 1 kudos
Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project
Hi everyone,I’m working through Databricks deployment and orchestration concepts and wanted to understand how Databricks Asset Bundles and Lakeflow Jobs fit together.Real-world scenario:Imagine an e-commerce company has a customer data pipeline that ...
- 426 Views
- 4 replies
- 1 kudos
- 707 Views
- 6 replies
- 1 kudos
Lakeflow connect Ingestion pipeline notification for gateway pipeline
Hi Guys,As you guys know that when we are building lake flow connect ingestion pipeline in UI. The pipeline consist both gateway pipeline and ingestion pipeline together. We have notification for the ingestion pipeline but not for gateway pipeline. I...
- 707 Views
- 6 replies
- 1 kudos
- 1 kudos
@srikanthp24 As mentioned earlier, The UI currently doesn't allow to configure notifications. The work arounds suggested are :DAB - In Asset bundles, can declare the Yaml constructs with all the config along with notifications and deploy via CI/CD. ...
- 1 kudos
- 825 Views
- 6 replies
- 1 kudos
VS Code and Connect - Manual Environment Setup
The current version of the extension forces usage of certain python packaging / project tools. I used to be able to use the extension with my project which uses pixi, by setting up the dependencies myself in pyproject.toml / pixi.toml.The current ver...
- 825 Views
- 6 replies
- 1 kudos
- 1 kudos
Databricks Connect is just a Python package, so you can bypass the extension entirely and keep your pixi setup.Add databricks-connect to your pixi environment, select that interpreter as your kernel in VS Code, and create the session yourself:from d...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
