- 375 Views
- 2 replies
- 0 kudos
1.Not able to access interent (gcp databricks) 2.both ingress and egress access given
1.Not able to access interent (gcp databricks) 2.both ingress and egress access given in default policy2.How to overcome this issue and i am using serverless3.in free databricks able to use interent in gcp based databricks it is creating issueimport...
- 375 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Hariharan_0510 , I took a look at both internal and external documentation and here is what I found. Building on @balajij8's reply, the serverless egress policy is the right place to start, but two things before you go hunting further. First, ...
- 0 kudos
- 2438 Views
- 4 replies
- 0 kudos
Resolved! Slow writes to managed volume
Hi All, I am using managed volumes as an intermediate storage to write a decrypted file before moving to data lake storage. Strangely the write operation is taking a lot of time (22 mins) to write a small file to volumes and it takes only few seconds...
- 2438 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi All, Thank you for you advice regarding this issue, but the root cause was something else. We had 2 private endpoints for ADLS (BLOB and DFS), and the firewall was only open for one BLOB PE. As a process Databricks first tries to resolve connectio...
- 0 kudos
- 542 Views
- 3 replies
- 2 kudos
Databricks AI/BI Get Session User
Hello everyone,I am currently working with Databricks AI/BI Dashboards and I have a specific requirement regarding user context.I need to dynamically capture the "session user" (the person who is currently viewing the dashboard) rather than the "exec...
- 542 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi,Short answer: No. There's no built-in function or parameter that returns the viewer's identity while a dashboard runs on the publisher's credentials.Why: When you publish with Share data permissions (the default, formerly called "Embed credentials...
- 2 kudos
- 905 Views
- 6 replies
- 5 kudos
Resolved! org.apache.iceberg.connect.IcebergSinkConnector to sink data from kafka to databricks - large volume
Hey.We have different number of topics per let's call it "subject". The connector works great for us if there are not so many topics (partitions) in subject. We have issue when we try to sink 1500 topics, 3 partitions each. We've sharded the topics i...
- 905 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello All.Thank you all for help. Here is what worked for us:## Original symptomsEach connector started normally and consumed records for more than an hour. Kafka Connect reportedthe connector and tasks as `RUNNING`, and source-topic lag decreased. T...
- 5 kudos
- 17737 Views
- 10 replies
- 8 kudos
The log in function and password reset function in the community edition do not work
I want to register a databrick account. I already set up my account. I also receive the email to set my password. However, I cannot use my password to log in the community account. I can use it to log in my standard account. I also click the reset th...
- 17737 Views
- 10 replies
- 8 kudos
- 8 kudos
Databricks guys claim at 190 Billion valuation ? or more ??Still struggling to send password reset email ??handover the project to me, I can create better customer experience!!! No team needed!!https://www.webassessor.com/#/twForgotPassword is the wo...
- 8 kudos
- 245 Views
- 0 replies
- 0 kudos
Built my first Databricks App with Lakebase
Screenshot of my deployed app attached. Thanks for the great quest!#Databricks #Lakebase #AppKit
- 245 Views
- 0 replies
- 0 kudos
- 400 Views
- 1 replies
- 0 kudos
Tip: Why your Delta MERGE fails with "multiple source rows matched" (and the simple fix)
Hi everyone If you run incremental loads with MERGE INTO on Delta tables, you may have seen a merge fail with an error saying multiple source rows matched the same target row.Why it happensA MERGE compares each row in your source batch to your target...
- 400 Views
- 1 replies
- 0 kudos
- 0 kudos
Great breakdown! That exact error has tripped up many a CDC pipeline. When working in PySpark, I typically use a similar window approach with row_number() and a strict tiebreaker like a sequence number before registering the temporary view for the me...
- 0 kudos
- 1096 Views
- 2 replies
- 3 kudos
Resolved! Designing an Effective "Quarantine" Pattern for Failed DLT Expectations
Hi everyone,I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.While expect_or_drop and expect_or_fail are straightforward, I’m struggling to fi...
- 1096 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @Khasim_1, this was an interesting one and I wanted to experiment a bit. I put this together if it helps - https://medium.com/lab-notes/three-lakeflow-quarantine-designs-tested-and-the-one-condition-every-queue-needs-1bea1c1588ff
- 3 kudos
- 852 Views
- 3 replies
- 2 kudos
Resolved! Liquid clustering vs. partitioning for a 5 TB Silver table with frequent MERGEs
Hi all,We have a Silver Delta table (~5 TB, growing ~20 GB/day) that is updated every hour with a MERGE on order_id. Right now it's partitioned by order_date. Most MERGE batches touch the last 3 days, but late updates can reach orders up to 90 days o...
- 852 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi! Good candidate for liquid clustering, since the "hot 3 days plus a long tail of late updates" pattern is exactly where fixed date partitions start to hurt. A few thoughts on each question.MERGE performanceResults vary a lot by data distribution, ...
- 2 kudos
- 827 Views
- 2 replies
- 3 kudos
Resolved! Secrets Redaction seems to work only when the retrieved value is assigned to a varible
I randomly encountered the following behavior when setting up Big Query federation in Databricks Free Edition and wondered if this is expected behavior or just something that happens in the Executing cell 1 as captured in the screenshot prints out th...
- 827 Views
- 2 replies
- 3 kudos
- 3 kudos
hi, this is expected behaviour and is not specific to Free Edition. Databricks documents that a secret assigned through dbutils.secrets.get() and subsequently printed should be replaced with [REDACTED]. Redaction is best-effort and applies to literal...
- 3 kudos
- 721 Views
- 5 replies
- 1 kudos
Connecting to Snowflake via Workload Identity Federation
Hi everyone,I appreciate your time!! We currently use username and password authentication to connect to Snowflake from a Databricks Spark job. We are exploring Workload Identity Federation as an alternative so that we can eliminate long-lived Snowfl...
- 721 Views
- 5 replies
- 1 kudos
- 1 kudos
Great topic—WIF can simplify Snowflake authentication quite a bit. Curious if anyone has a clean setup that works reliably with Databricks!
- 1 kudos
- 1112 Views
- 6 replies
- 6 kudos
Best Architecture Approach for Databricks Multi-Cloud and Multi-Region Deployment
I am working on a data platform requirement where we need to support multiple cloud providers and multiple regions using Databricks.For example:AWS – Region 1AWS – Region 2Azure – Region 1Azure – Region 2The requirement is to maintain data processing...
- 1112 Views
- 6 replies
- 6 kudos
- 6 kudos
@Srini_Pesala You can deploy one workspace per region per cloud provider. Unity Catalog enforces certain limits on metastore per region. Data residency is handled natively through Databricks Geos and it ensures customer data is strictly processed and...
- 6 kudos
- 1826 Views
- 5 replies
- 0 kudos
Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster
Hi everyone, I’m facing an issue when running a notebook on a Databricks All-purpose cluster. Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.I tried setting spar...
- 1826 Views
- 5 replies
- 0 kudos
- 0 kudos
Greetings @zenwanderer, I did some digging and here is what I found. Your follow-up narrows the ask: you want the workspace, not each developer, to catch and kill any interactive cell that runs past a threshold. Short answer: there's no supported, bu...
- 0 kudos
- 896 Views
- 6 replies
- 8 kudos
CSV export from AI/BI Dashboard table widget does not preserve visual column order
SummaryWhen downloading data as CSV from a table widget on an AI/BI Dashboard (kebab menu → Download → Data → Download CSV), the column order in the exported file does not match the column order displayed in the dashboard. All aggregated (measure) co...
- 896 Views
- 6 replies
- 8 kudos
- 8 kudos
This issue occurs if your dashboard has a semantic model attached i.e., you've clicked "Get started with relationships" even if it doesn't have any relationships defined. Just having the semantic model present is enough to trigger it.The problem is i...
- 8 kudos
- 778 Views
- 2 replies
- 3 kudos
Resolved! Managing "Secret" Injection in DABs across Dev/Stage/Prod
Hi everyone, I am fully moving our team to Databricks Asset Bundles (DABs) for CI/CD, but I’m struggling with the "Secret" management pattern.How are you handling the injection of secrets (like API keys for ingestion) into DABs without hardcoding any...
- 778 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,The basic rule: the bundle manages secret scopes and references to secrets, never the secret values.1. Keeping secrets out of databricks.yml: Only reference scope and key names. Jobs resolve the secret at runtime:In code: dbutils.secrets.get(scope...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
