cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Davide
by • New Contributor III
  • 873 Views
  • 2 replies
  • 1 kudos

Resolved! SET MANAGED AND ROLLBACK IN RELATION TO VACUUM

Hello everyone,I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE table_name SET MANAGED.Databricks documentation states that the conversion can be rolled back with...

  • 873 Views
  • 2 replies
  • 1 kudos
Latest Reply
DoTA
Honored Contributor
  • 1 kudos

Hi, I could not find these three cases documented either, so I will separate what the docs say from what I would treat as an assumption. What is documented (Convert external tables to managed): Databricks keeps the data in the old external location f...

  • 1 kudos
1 More Replies
srikanthp24
by • New Contributor III
  • 432 Views
  • 1 replies
  • 0 kudos

Regarding the DAB Ingestion Pipeline Creating the Volume

Hello Guys,By taking the below yaml code as a reference I have created the databrick asset bundle for data transfer from sql server to databricks. But with below code the staging volume getting created by gateway pipeline automatically, which unable ...

srikanthp24_0-1790135779415.png
  • 432 Views
  • 1 replies
  • 0 kudos
Latest Reply
DoTA
Honored Contributor
  • 0 kudos

Hi, a few things that may unblock you here. 1) Custom volume name. In gateway_definition you can set gateway_storage_catalog, gateway_storage_schema and gateway_storage_name. If you leave gateway_storage_name out, the gateway auto-generates a volume ...

  • 0 kudos
Islam_hoti
by • Contributor
  • 886 Views
  • 3 replies
  • 5 kudos

Resolved! Photon enabled but a large share of the plan is falling back, cost up and runtime flat

Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...

  • 886 Views
  • 3 replies
  • 5 kudos
Latest Reply
Khasim_1
Contributor
  • 5 kudos

Hi @Islam_hoti ,Photon is fantastic, but it’s an 'all-or-nothing' value proposition. Once you hit a fallback, you lose the vectorized execution benefit for that entire sub-tree, and you're still paying the DBU premium. Here is how we approach auditin...

  • 5 kudos
2 More Replies
Hariharan_0510
by • New Contributor
  • 449 Views
  • 2 replies
  • 0 kudos

1.Not able to access interent (gcp databricks) 2.both ingress and egress access given

1.Not able to access interent (gcp databricks) 2.both ingress and egress access given in default policy2.How to overcome this issue and  i am using serverless3.in free databricks able to use interent in gcp based databricks it is creating issueimport...

  • 449 Views
  • 2 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Hello @Hariharan_0510 , I took a look at both internal and external documentation and here is what I found. Building on @balajij8's reply, the serverless egress policy is the right place to start, but two things before you go hunting further. First, ...

  • 0 kudos
1 More Replies
yashojha
by • New Contributor III
  • 2506 Views
  • 4 replies
  • 0 kudos

Resolved! Slow writes to managed volume

Hi All, I am using managed volumes as an intermediate storage to write a decrypted file before moving to data lake storage. Strangely the write operation is taking a lot of time (22 mins) to write a small file to volumes and it takes only few seconds...

yashojha_1-1771488017215.png yashojha_3-1771488035100.png
  • 2506 Views
  • 4 replies
  • 0 kudos
Latest Reply
yashojha
New Contributor III
  • 0 kudos

Hi All, Thank you for you advice regarding this issue, but the root cause was something else. We had 2 private endpoints for ADLS (BLOB and DFS), and the firewall was only open for one BLOB PE. As a process Databricks first tries to resolve connectio...

  • 0 kudos
3 More Replies
mmsanteago
by • New Contributor II
  • 697 Views
  • 3 replies
  • 2 kudos

Databricks AI/BI Get Session User

Hello everyone,I am currently working with Databricks AI/BI Dashboards and I have a specific requirement regarding user context.I need to dynamically capture the "session user" (the person who is currently viewing the dashboard) rather than the "exec...

  • 697 Views
  • 3 replies
  • 2 kudos
Latest Reply
Islam_hoti
Contributor
  • 2 kudos

Hi,Short answer: No. There's no built-in function or parameter that returns the viewer's identity while a dashboard runs on the publisher's credentials.Why: When you publish with Share data permissions (the default, formerly called "Embed credentials...

  • 2 kudos
2 More Replies
dawidch
by • New Contributor III
  • 1102 Views
  • 6 replies
  • 5 kudos

Resolved! org.apache.iceberg.connect.IcebergSinkConnector to sink data from kafka to databricks - large volume

Hey.We have different number of topics per let's call it "subject". The connector works great for us if there are not so many topics (partitions) in subject. We have issue when we try to sink 1500 topics, 3 partitions each. We've sharded the topics i...

  • 1102 Views
  • 6 replies
  • 5 kudos
Latest Reply
dawidch
New Contributor III
  • 5 kudos

Hello All.Thank you all for help. Here is what worked for us:## Original symptomsEach connector started normally and consumed records for more than an hour. Kafka Connect reportedthe connector and tasks as `RUNNING`, and source-topic lag decreased. T...

  • 5 kudos
5 More Replies
self-employed
by • Contributor
  • 17897 Views
  • 10 replies
  • 8 kudos

The log in function and password reset function in the community edition do not work

I want to register a databrick account. I already set up my account. I also receive the email to set my password. However, I cannot use my password to log in the community account. I can use it to log in my standard account. I also click the reset th...

  • 17897 Views
  • 10 replies
  • 8 kudos
Latest Reply
snemmal
New Contributor II
  • 8 kudos

Databricks guys claim at 190 Billion valuation ? or more ??Still struggling to send password reset email ??handover the project to me, I can create better customer experience!!! No team needed!!https://www.webassessor.com/#/twForgotPassword is the wo...

  • 8 kudos
9 More Replies
LiresaFerizaj
by • Contributor
  • 500 Views
  • 1 replies
  • 0 kudos

Tip: Why your Delta MERGE fails with "multiple source rows matched" (and the simple fix)

Hi everyone If you run incremental loads with MERGE INTO on Delta tables, you may have seen a merge fail with an error saying multiple source rows matched the same target row.Why it happensA MERGE compares each row in your source batch to your target...

  • 500 Views
  • 1 replies
  • 0 kudos
Latest Reply
monika532lara
New Contributor II
  • 0 kudos

Great breakdown! That exact error has tripped up many a CDC pipeline. When working in PySpark, I typically use a similar window approach with row_number() and a strict tiebreaker like a sequence number before registering the temporary view for the me...

  • 0 kudos
Khasim_1
by • Contributor
  • 1246 Views
  • 2 replies
  • 3 kudos

Resolved! Designing an Effective "Quarantine" Pattern for Failed DLT Expectations

 Hi everyone,I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.While expect_or_drop and expect_or_fail are straightforward, I’m struggling to fi...

  • 1246 Views
  • 2 replies
  • 3 kudos
Latest Reply
gnakan
Databricks MVP
  • 3 kudos

Hey @Khasim_1, this was an interesting one and I wanted to experiment a bit. I put this together if it helps - https://medium.com/lab-notes/three-lakeflow-quarantine-designs-tested-and-the-one-condition-every-queue-needs-1bea1c1588ff

  • 3 kudos
1 More Replies
Islam_hoti
by • Contributor
  • 1001 Views
  • 3 replies
  • 2 kudos

Resolved! Liquid clustering vs. partitioning for a 5 TB Silver table with frequent MERGEs

Hi all,We have a Silver Delta table (~5 TB, growing ~20 GB/day) that is updated every hour with a MERGE on order_id. Right now it's partitioned by order_date. Most MERGE batches touch the last 3 days, but late updates can reach orders up to 90 days o...

  • 1001 Views
  • 3 replies
  • 2 kudos
Latest Reply
LiresaFerizaj
Contributor
  • 2 kudos

Hi! Good candidate for liquid clustering, since the "hot 3 days plus a long tail of late updates" pattern is exactly where fixed date partitions start to hurt. A few thoughts on each question.MERGE performanceResults vary a lot by data distribution, ...

  • 2 kudos
2 More Replies
irfanghat
by • New Contributor III
  • 915 Views
  • 2 replies
  • 3 kudos

Resolved! Secrets Redaction seems to work only when the retrieved value is assigned to a varible

I randomly encountered the following behavior when setting up Big Query federation in Databricks Free Edition and wondered if this is expected behavior or just something that happens in the Executing cell 1 as captured in the screenshot prints out th...

  • 915 Views
  • 2 replies
  • 3 kudos
Latest Reply
tom_n
Databricks Employee
  • 3 kudos

hi, this is expected behaviour and is not specific to Free Edition. Databricks documents that a secret assigned through dbutils.secrets.get() and subsequently printed should be replaced with [REDACTED]. Redaction is best-effort and applies to literal...

  • 3 kudos
1 More Replies
SKiruthiga
by • New Contributor II
  • 902 Views
  • 5 replies
  • 1 kudos

Connecting to Snowflake via Workload Identity Federation

Hi everyone,I appreciate your time!! We currently use username and password authentication to connect to Snowflake from a Databricks Spark job. We are exploring Workload Identity Federation as an alternative so that we can eliminate long-lived Snowfl...

  • 902 Views
  • 5 replies
  • 1 kudos
Latest Reply
ThiamLee
Contributor III
  • 1 kudos

Great topic—WIF can simplify Snowflake authentication quite a bit. Curious if anyone has a clean setup that works reliably with Databricks!

  • 1 kudos
4 More Replies
Srini_Pesala
by • Databricks Partner
  • 1420 Views
  • 6 replies
  • 6 kudos

Best Architecture Approach for Databricks Multi-Cloud and Multi-Region Deployment

I am working on a data platform requirement where we need to support multiple cloud providers and multiple regions using Databricks.For example:AWS – Region 1AWS – Region 2Azure – Region 1Azure – Region 2The requirement is to maintain data processing...

  • 1420 Views
  • 6 replies
  • 6 kudos
Latest Reply
balajij8
Esteemed Contributor III
  • 6 kudos

@Srini_Pesala You can deploy one workspace per region per cloud provider. Unity Catalog enforces certain limits on metastore per region. Data residency is handled natively through Databricks Geos and it ensures customer data is strictly processed and...

  • 6 kudos
5 More Replies
Labels