Databricks Community

Hello @sachin_kanchan! I understand the frustration, and I appreciate you sharing your experience. The Community Edition is meant to provide a smooth experience, and this shouldn’t be happening. We usually ask users to drop an email to help@databrick...

We recommend using spatial frameworks to speed up things like spatial joins, point-in-polygon, etc, like databricks mosaic or apache sedona. Without these frameworks, many of these operations result in unoptimized and explosive crossjoins.

Did you find any solution?

For datasets with constantly changing schemas, we recommend using the Variant type.

Here are some strategies for debugging this: Before you perform each merge, write your source dataframe out as a table, and include the target table's version in the table's nameIf possible, enable the change data feed on your table so as to see chan...

Hi @sachamourier, Could you please clarify what is the inconsistency? are some packages missing or the incorrect library was loaded?

Hi @LeenB For example each cell execution you can build up with try except command. Example belowtry:     print("Hello world")    #your code of each cellexcept Exception as e:    print("Issue with printing hello world")For sure it is not recommended ...

For me it is working at 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)

You cannot use Spark session explicitly inside Executor logic. Here you are trying mapPartitions which makes the customlogic to get executed inside the executor thread. Either you need to change whole problem approach to segregate spark variable usag...

Hi @Alberto_Umana , Just a QQ would we be able to change table properties like adding column details, column tagging and Column level masking on the snowflake tables that are under the foreign catalog created?