Hello,I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.So, as per the Microsoft migration pre-requisites:File or directory names with only spaces or tabs, ending with a ., containing ...
HiWhen I try to spin up a cluster, it gives me a bootstrap timeout error{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-00b2b7acdd82e5fde), status: INSTANCE_INITIALIZING, workerEnv...
Thanks guys. It's indeed a network issue on the AWS side. It's resolved now
Hello,I have a single column dataframe and I want to transform the content into a stringEG df=abcdefxyzToabc, def, xyz Thanks
sure: %pythonfrom pyspark.sql.functions import from_json, col, concat_wsfrom pyspark.sql.types import *schema = StructType([StructField('meterDateTime', StringType(), True), StructField('meterId', LongType(), True), StructField('meteringState', Strin...
Hi, I have a set of values to be searched from a table, for which i was trying to assign them to a variable first and then trying to use the variable in spark.sql, but i'm unable to fetch the records. Please see the image attached and correct my code...
Hi, One way to address the example provided in your screenshot is by using a combination of a python f-string and a Common Table Expression like shown below. This is assuming that in reality the two tables are different unlike in the provided screens...
Hello, I'm using a DLT pipeline with expectationsexpect_or_drop(...) To test it, I added files that contain records that should be dropped, and indeed when running the pipeline I can see some rows were dropped.However when looking at the DLT Event lo...
Hello @erigaud The issue appears to be related to the details.flow_progress.data_quality.dropped_records field always being 0, despite records being dropped. This might be because the expect_or_drop operator isn't updating the dropped_records field ...
I am trying to get the Bigquery data to Databricks using Notebooks. Following the steps based on this https://docs.databricks.com/external-data/bigquery.html. I believe I am making some mistake with this step and getting the below error. I tried givi...
Hi!Did you get the problem solved?I am facing the same issueplease guide
Hi Team,I would like to know how we can continue streaming change data feed from a delta table when its schema is changed ( non-additive schema changes like drop/rename column / schema migration ).I came across schemaTrackingLocation in readStream bu...
Hi!I'm converting an existing TSQL script into Databricks SQL.In TSQL, the below script returns 1select datepart(WEEK,'2022-01-01')In Databricks SQL, then below script returns 52.select date_part('week','2022-01-01') Does Databricks SQL have somethin...
This worked for me.SELECT your_date_column, CASE WHEN DAYOFYEAR(your_date_column) <= 7 THEN 'Week 1' ELSE 'Week ' || CAST(CEIL((DAYOFYEAR(your_date_column) - 7) / 7.0) + 1 AS STRING) END AS WeekNumberFROM your_date_table;
Hi all,I'm attempting to switch our DLT pipeline using Auto Loader from Directory Listing to File Notification mode, and running in to S3 Access Denied issues with very little detail. I have followed all the instructions here and here to set up File ...
Thanks @Retired_mod, we ended up abandoning this route due to limitations imposed by the Shared compute access mode enforced by DLT's, and opted for a standard Spark Structured Streaming Job (using Kafka) in the end.
Job is scheduled on interactive cluster, and it failed with below error and in the next scheduled run it ran fine. I want to why this error occurred and how can I prevent from occurring this again.How to debug these types of error? com.databricks.b...
@JKR Could you try setting the configurations below at the cluster level and retry the job?spark.databricks.python.defaultPythonRepl pythonshellspark.databricks.pyspark.py4j.pinnedThread false
Is there a way to mount Workspace folder (WSFS) to the Docker container if I'm using the Databricks Container Services ofr running a general purpose cluster?If I create a cluster without a Docker image, the `!ls` command in Databricks notebook return...
Hello:Thanks for contacting Databricks Support! I'm afraid that mounting the WSFS directly into a Docker container isn't directly supported. The Databricks workspace is a specialized environment and isn't directly analogous to a regular filesystem. W...
Databricks Certified Associate Developer for Apache Spark 3.0
Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it
Hi allHaving some fun trying to run a notebook on a shared UC-aware, shared cluster - I keep on running into this error:py4j.security.Py4JSecurityException: Method public static org.apache.spark.sql.SparkSession org.apache.sedona.spark.SedonaContext....
Started getting this error while running all the scripts. All the scripts were running fine before. I tried de-attaching and also restart nothing seems to work.Internal error. Attach your notebook to a different cluster or restart the current cluste...
Hi @210573 (Customer)​ ,I got the same error, tried to restart and create a new cluster but the solution does not work. What I did to fix the issue: Instead of putting in function, break the code out to run line by line. I just want to see where the ...
Hello all, I am following the Databrick's documentation on unit testing found here: Run tests with pytest for the Databricks extension for Visual Studio Code - Azure Databricks | Microsoft LearnHowever, when taking it a step further I get an ImportEr...
HelloImport errors happen often with Pytest. To Debug this error you can add this in your "test_myfunction_test.py":import sys # printing all directories for # interpreter to search sys.pathsys.path is a built-in variable within the sys module. I...
| User | Count |
|---|---|
| 1628 | |
| 790 | |
| 510 | |
| 349 | |
| 287 |
