- 8837 Views
- 5 replies
- 0 kudos
Optimize & Compaction
Hi,From which data bricks runtime will support Optimize and compaction
- 8837 Views
- 5 replies
- 0 kudos
- 0 kudos
Optimize and compaction are operations commonly used in Apache Spark for optimizing and improving the performance of data storage and processing. Databricks, which is a cloud-based platform for Apache Spark, provides support for these operations on v...
- 0 kudos
- 5061 Views
- 6 replies
- 1 kudos
PGP encryption and decryption using gnupg
Hi,We are using python-gnupg==0.4.8 package for encryption and decryption and this was working as expected when we are using Databricks runtime : 9.1 LTS but when we upgarded our runtime to 12.1, it stopped working with error "gnupghome should be a d...
- 5061 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Anjum Aara​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
- 1 kudos
- 8617 Views
- 8 replies
- 10 kudos
Resolved! New strange error on Runtime 12 and above: java.lang.AssertionError: assertion failed
Hi allI struggle to find out why this error message suddenly pops up after running a cell in a notebook. The notebook is trying to run a simple "INSERT INTO" command in SQL. When I only do a SELECT clause, the cell runs without error. Also, I only ge...
- 8617 Views
- 8 replies
- 10 kudos
- 10 kudos
Thanks for reporting! We have identified a defect with an early version of DBR 12 related to INSERT INTO .. SELECT when certain query patterns are involved. The defect has since been fixed. Please let us know if you have any additional questions.
- 10 kudos
- 1487 Views
- 1 replies
- 6 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 1487 Views
- 1 replies
- 6 kudos
- 6 kudos
Thank you for sharing @Hubert Dudek​ !!!
- 6 kudos
- 7436 Views
- 0 replies
- 2 kudos
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notesMarch 13 -...
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesMarch 13 - 17, 2023Execute SQL cells in the notebook in parallelYou can now run SQL cells in Databricks noteboo...
- 7436 Views
- 0 replies
- 2 kudos
- 1076 Views
- 0 replies
- 5 kudos
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and...
Exciting news for #azure users! The #databricks runtime 12.2 has been officially released as a long-term support (LTS) version, providing a stable and reliable platform for users to build and deploy their applications. As part of this release, the en...
- 1076 Views
- 0 replies
- 5 kudos
- 1091 Views
- 1 replies
- 10 kudos
Since databricks runtime 12.1 "WHEN NOT MATCHED BY SOURCE" was added to MERGE syntax. For example, using that option, we can quickly delete ...
Since databricks runtime 12.1 "WHEN NOT MATCHED BY SOURCE" was added to MERGE syntax. For example, using that option, we can quickly delete all target rows which doesn't match any source.
- 1091 Views
- 1 replies
- 10 kudos
- 10 kudos
Thank you for sharing @Hubert Dudek​
- 10 kudos
- 2448 Views
- 2 replies
- 4 kudos
what is the supported mssql connector for Databricks runtime 11.3LTS Scala 2.12 Spark 3.3.0?
We were using mssql connector -com.microsoft.azure:spark-mssql-connector_2.12_3.0:1.0.0-alpha with 10.3LTS DBR. As we need to upgrade to higher version of DBR to make use of new functions like unpivot/melt in the notebooks. -com.microsoft.azure:spark...
- 2448 Views
- 2 replies
- 4 kudos
- 4 kudos
Is the spark 3.3 series even supported by the connector yet ?As per the [github link](https://github.com/microsoft/sql-spark-connector#current-releases) - assuming this is the library you are trying to use ?The latest Spark 2.4.x compatible connector...
- 4 kudos
- 1099 Views
- 1 replies
- 8 kudos
Since databricks runtime 12 and spark version 3.3.1, what you PIVOT can be easily UNPIVOT.Â
Since databricks runtime 12 and spark version 3.3.1, what you PIVOT can be easily UNPIVOT.
- 1099 Views
- 1 replies
- 8 kudos
- 8 kudos
Thanks for sharing
- 8 kudos
- 6614 Views
- 4 replies
- 2 kudos
Resolved! java.lang.NoSuchMethodError after upgrade to Databricks Runtime 11.3 LTS
Hi, I am using Databricks and want to upgrade to Databricks runtime version 11.3 LTS which uses Spark 3.3 now. Current system enviroment:Operating System: Ubuntu 20.04.4 LTSJava: Zulu 8.56.0.21-CA-linux64Python: 3.8.10Delta Lake: 1.1.0Target system ...
- 6614 Views
- 4 replies
- 2 kudos
- 14793 Views
- 14 replies
- 5 kudos
toPandas() causes IndexOutOfBoundsException in Apache Arrow
Using DBR 10.0When calling toPandas() the worker fails with IndexOutOfBoundsException. It seems like ArrowWriter.sizeInBytes (which looks like a proprietary method since I can't find it in OSS) calls arrow's getBufferSizeFor which fails with this err...
- 14793 Views
- 14 replies
- 5 kudos
- 5 kudos
I am also facing the same issue, I have applied the config: `spark.sql.execution.arrow.pyspark.enabled` set to `false`, but still facing the same issue. Any Idea, what's going on???. Please help me out....org.apache.spark.SparkException: Job aborted ...
- 5 kudos
- 1664 Views
- 3 replies
- 5 kudos
Weekly Release Notes Recap�� Here’s a quick recap of the latest release notes updates from the past one week. Databricks platform release notes...
Weekly Release Notes Recap Here’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notes December 1-6, 2022Partner Connect supports connecting to AtScale:You can now easily create a connection betwe...
- 1664 Views
- 3 replies
- 5 kudos
- 5 kudos
@Uma Maheswara Rao Desula​ if i am not wrong, below ideas portal should help you Ideas Portal | Databricks on AWS
- 5 kudos
- 2401 Views
- 4 replies
- 8 kudos
Databricks Runtime Support
How Long are Databricks runtimes supported for? How often are they updated?You can learn more about the Databricks runtime support lifecycle here (AWS | Azure | GCP).Long Term Support (LTS) runtimes are released every 6 months and supported for 2 yea...
- 2401 Views
- 4 replies
- 8 kudos
- 7480 Views
- 1 replies
- 5 kudos
java.util.NoSuchElementException: key not found
Hello,We are using a Azure Databricks with Standard DS14_V2 Cluster with Runtime 9.1 LTS, Spark 3.1.2 and Scala 2.12 and facing the below issue frequently when running our ETL pipeline. As part of the operation that is failing there are several joins...
- 7480 Views
- 1 replies
- 5 kudos
- 5 kudos
Hey man,Please use these configuration in your cluster and it will work,spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsString truespark.speculation falsespark.sql.legacy.timeParserPolicy LEGACYif it wont work let me know what problem...
- 5 kudos
- 3570 Views
- 8 replies
- 19 kudos
Databricks-connect not available on Databricks Runtime > 10.4
Hello Databricks Team,Databricks-connect doesn't work on databricks runtime 11.3.Databricks recommends that we use dbx for Databricks Lab instead of databricks-connect. Databricks plans no new feature development for Databricks Connect at this time.D...
- 3570 Views
- 8 replies
- 19 kudos
- 19 kudos
thx @Landan George​ do you have any ETA about the public preview ?
- 19 kudos