Databricks Community

Hi @Ashok_Vengala, This comes up fairly often, so let me break it down. the different behavior you are seeing between older and newer workspaces is expected, and I can help clarify what is happening here. UNDERSTANDING THE DEPRECATION The legacy Iceb...

Hi @ajay_wavicle, Good timing on this question. Connecting Azure storage accounts to Databricks using a User-Assigned Managed Identity is a great approach -- it avoids the need to manage secrets and supports storage firewall configurations. Here is a...

Hi @ajay_wavicle, Thanks for the detailed writeup. The reason you lose access to Spark when using %sh is that it launches a completely separate Linux process on the driver node. That process runs outside the notebook runtime, so it has no connection ...

Hi @JD18, Welcome to Databricks and Thanks for raising this. SCD Type 2 backfilling with streaming tables is a common need, and the good news is that the AUTO CDC framework (formerly APPLY CHANGES INTO) has built-in capabilities to handle this -- you...

Hi @turagittech, Great question and I understand the concern. The good news is that updating the notification email in your Asset Bundle YAML and redeploying should NOT recreate your gateway or ingestion pipeline resources. Let me explain why and wal...

Hi @kmcas10, Let me share some guidance on this. this is a scenario that comes up when transitioning from Databricks PVC (private cloud) to SaaS with legacy ODBC tooling, and there are several things you can try to bridge the gap until your full migr...

Hi @NishantTiwari, I see you have already upgraded to DBR 14.3+ but are still hitting the same SSL errors. That makes sense, and here is why: the two errors you are seeing point to the 3rd party server using weak or outdated SSL certificates, not an ...

Hi @QueryingQuail, Good question -- I can see from the follow-up discussion that you are looking for practical guidance that goes beyond what a generic AI prompt would give you -- specifically how to handle this across both managed connectors (like t...

Hi @DoredlaCharan, The root cause here is straightforward: setting sampleSize to 100,000 forces the MongoDB Spark Connector to pull 100K documents onto your driver node just for schema inference. With 100+ keys per document and mergeSchema enabled, t...

Hi @DylanStout, This is worth walking through carefully. It sounds like you have already done solid research on the constraints. Let me walk you through the most likely reason your init script is hanging and provide a complete working approach for of...

Hi @ravipal-global, I have seen this pattern before. The behavior you are seeing is expected. Let me explain why it happens and then walk through several approaches that can help you achieve delete-and-reload without requiring a full refresh of your ...

Hi @Pratikmsbsvm, Happy to help with this one. SAP S/4HANA to Databricks is one of the most common enterprise data migration scenarios, and there are several well-proven approaches depending on your requirements for data freshness, volume, and budget...

Hi @bunny_9090, Let me walk you through this. Your analysis of the root cause is spot on. Let me expand on what is happening and walk through the recommended approaches to address it. WHY THIS HAPPENS -- IEEE 754 FLOATING-POINT REPRESENTATION Both FL...