- 1555 Views
- 1 replies
- 0 kudos
Can not able to connect to ADLS Gen2 using ABFSS (urgent)
Hi all,I was having an error when trying to connect to azure data lake. "SSLHandshakeExceptio" Please see the detail below.Please help!
- 1555 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message you provided, "SSLHandshakeExceptio," indicates that there is a problem with the SSL handshake between your system and the Azure Data Lake. This issue can be caused by various factors, such as incorrect settings, network issues, or ...
- 0 kudos
- 1913 Views
- 0 replies
- 0 kudos
Structured Streaming - Kafka Offset Management
In my team, we decided to move from spark streaming to structured streaming, mainly cause it says that it's legacy and we want to benefit new features from structured streaming.However we have an issue with committing offsets.Previously on spark stre...
- 1913 Views
- 0 replies
- 0 kudos
- 21684 Views
- 2 replies
- 0 kudos
.overwriteschema + writestream
HelloI have issue with overwriting schema while using writestream - I do not receive any error - however schema remain unchangedBelow exampledf_abc = spark.readstream .format("cloudFiles") .option("cloudFiles.format", "parquet") .option"cloudF...
- 21684 Views
- 2 replies
- 0 kudos
- 10225 Views
- 3 replies
- 0 kudos
dataset.cache() not working : NoSuchObjectException(message:There is no database named global_temp)
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)at org.apache.hadoop.hive.metastore.ObjectStore.getMDatabase(ObjectStore.java:508)at org.apache.hadoop.hive.metastore.ObjectStore.getDatabase(ObjectStore.j...
- 10225 Views
- 3 replies
- 0 kudos
- 1451 Views
- 0 replies
- 0 kudos
I want to perform transpose in my dataframe using pyspark
I have 26 columns and 18k rows which I want to transpose and make 18k columns and 26 rows.Don't want any data summation of aggregation just transpose as it is. Can anyone pls suggest.P.S Not via converting to pandas because of large data
- 1451 Views
- 0 replies
- 0 kudos
- 2445 Views
- 1 replies
- 0 kudos
Delta Live Tables Slowly Changing Dimensions Type 2 with Joins
Hi,I may be missing something really obvious here. The organisation I work for has started using Delta Live Tables in Databricks for data modelling, recently. One of the dimensions I am trying to model takes data from 3 existing tables in our data la...
- 2445 Views
- 1 replies
- 0 kudos
- 0 kudos
Can it be because the default join is `inner` and that means the row must exists in both tables
- 0 kudos
- 1222 Views
- 1 replies
- 0 kudos
Databricks Community Post Editor Issues
Anyone else constantly having errors with this editor when using any of the 'features' like code sample?Can we please have a Markdown Editor or at least the ability to edit the HTML this tool creates to fix all the bugs it makes?
- 1222 Views
- 1 replies
- 0 kudos
- 0 kudos
Here is a fun one: "The message body contains h d, which is not permitted in this community. Please remove this content before sending your post."Had to add the space between h and d to be able to post it. This means code samples can't contain `ch d...
- 0 kudos
- 2173 Views
- 0 replies
- 0 kudos
Databricks Advanced Data Engineering Course Factually Incorrect and Misleading
On Video 4 of the Advanced Data Engineering with Databricks course at 3:08 the presenter says 'No one else can do what we can with a single solution' . This is far from truth, Palantir foundry is miles ahead of databricks in Data Governance , Ease of...
- 2173 Views
- 0 replies
- 0 kudos
- 6602 Views
- 0 replies
- 0 kudos
Error handling best practices
Hi Team,Could you please share the best practices for error handling in Databricks for the following: 1. Notebook level 2.Job level 3. Code level(Python) 4. streaming 5. DLT & Autoloader Kindly suggest details around Error handling...
- 6602 Views
- 0 replies
- 0 kudos
- 4757 Views
- 2 replies
- 1 kudos
Resolved! Rendering markdown images hard coded as data image png base64 in notebook
Hi all,For training purposes, I have cloned a repo from John Snow Labs into my Databricks account and am working in the notebook that you can review at https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/open-source-nlp/03.0.SparkNLP_Pretr...
- 4757 Views
- 2 replies
- 1 kudos
- 1 kudos
Try changing the magic command for that cell from %md to %md-sandbox to see if that helps the image to render appropriately.
- 1 kudos
- 971 Views
- 0 replies
- 0 kudos
Type 2 - Slowly Changing Dimensions (SCD) without Streaming Tables?
Is there a way for me to get Type 2 SCD changes without using streaming tables?I'm worried streaming tables may have limitations that interfere with adoption.
- 971 Views
- 0 replies
- 0 kudos
- 3056 Views
- 1 replies
- 1 kudos
Tutorial "Query Data from a Notebook": access errors
I am trying to work through Tutorial: Query data from a Notebook.Access errors are defeating my attempts. Steps to reproduce:sign up for free trial through Databricks website. The path skipped the subscription-selection step and defaulted the trial t...
- 3056 Views
- 1 replies
- 1 kudos
- 1 kudos
Done, thanks!
- 1 kudos
- 1120 Views
- 0 replies
- 0 kudos
MLflow CVE-2023-6015 (vulnerabilities)
Hi,what mlflow version is used for the "Databricks Managed MLflow"Is there a workarround for MLflow CVE-2023-6015 (vulnerabilities) ?RegardsTimo
- 1120 Views
- 0 replies
- 0 kudos
- 1393 Views
- 0 replies
- 0 kudos
run spark-sql-perf from a notebook
I am looking to run spark-sql-perf on a databricks cluster I am not sure how to create a session that add spark-sql-perf jar dependencies
- 1393 Views
- 0 replies
- 0 kudos
- 1832 Views
- 1 replies
- 0 kudos
Internal error: Attach your notebook to a different compute or restart the current compute.
I am currently using a personal computer cluster [13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)] on GCP attached to a notebook. After running a few command lines without an issue, I end up getting this errorInternal error. Attach your notebook t...
- 1832 Views
- 1 replies
- 0 kudos
