Databricks

Hi @Jennifer,  Since TimestampNTZType is not currently supported for optimization, you can try a workaround by converting the timestamp column to a different data type before running the OPTIMIZE command.For example, you could convert the timestampNt...

Hi @jp_allard,  One approach to resolve this is to disable SSL certificate verification. However, keep in mind that this approach may compromise security.In your Databricks configuration file (usually located at ~/.databrickscfg), add the following l...

Hi @pernilak, It’s great that you’re using Databricks with Visual Studio Code (VSCode) for your development workflow! Let’s address the limitations you’ve encountered when working with files from Unity Catalog using native Python. When running Python...

Hi @jp_allard ,  The Unity Catalog is a newer feature in Databricks, designed to replace the traditional Hive Metastore.When transitioning from Hive Metastore to Unity Catalog, there might be differences in behavior due to underlying architectural ch...

We still face the problem (UC enabled shared cluster). Is there any resolution? @Kaniz  

How about not specifying the format?  This will already match common formats.When you still have nulls, you can use your list with known exotic formats.Another solution is working with regular expressions.  looking for 2 digit numbers not larger than...

Hi @Phuonganh, This error is not directly related to the Databricks SDK, but rather a misunderstanding of how to pass parameters in your SQL query. The param dictionary you’ve defined seems to have a typo in the value for the ‘a’ parameter. It should...

if you set .option("inferSchema", "false") all columns will be read as string.You will have to cast all the other columns to their appropriate type though.  So passing a schema seems easier to me.

Hi @zmsoft,  The processing power of the nodes running your Dlt pipeline matters. Using more powerful node types can significantly impact performance.Consider using a more robust node type, such as the Standard_E16ds_v4 or Standard_E32ds_v4.

As Spark is lazy evaluated, using only small clusters for read and large ones for writes is not something that will happen.The data is read when you apply an action (write f.e.).That being said:  I have no knowledge of a bug in Databricks on clusters...

I doubt it is possible.The driver runs the program, and sends tasks to the executors.  But since creating the ssh tunnel is no spark task, I don't think it will be established on any executor.

Hello,It is possible to have the expected result with dbutils.fs.cp("Origin location", "Destination location", True) and then create the table with the LOCATION of the Destination locationHope this helps

I don't think that it is possible at the moment.  UC focuses on cloud data.You might want to try to use Minio, but apparently UC does not support Minio yetPity, because that would be an awesome solution.

Thank you @Kaniz for your detailed explanation