- 5513 Views
- 2 replies
- 0 kudos
Python UDF in Unity Catalog - spark.sql error
I'm trying to utilise the option to create UDFs in Unity Catalog. That would be a great way to have functions available in a fairly straightforward manner without e.g. putting the function definitions in an extra notebook that I %run to make them ava...
- 5513 Views
- 2 replies
- 0 kudos
- 0 kudos
I came across the same problem. inside unity catalog UDF creation, spark.sql or spark.table doesn't work.Adding from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() into the session doesn't work as wellDon't know how to sol...
- 0 kudos
- 30391 Views
- 9 replies
- 5 kudos
Resolved! Getting "Job aborted due to stage failure" SparkException when trying to download full result
I have generated a result using SQL. But whenever I try to download the full result (1 million rows), it is throwing SparkException. I can download the preview result but not the full result. Why ? What happens under the hood when I try to download ...
- 30391 Views
- 9 replies
- 5 kudos
- 5 kudos
Job aborted due to stage failure: Task 6506 in stage 46.0 failed 4 times, most recent failure: Lost task 6506.3 in stage 46.0 (TID 12896) (10.**.***.*** executor 12): java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (a...
- 5 kudos
- 7138 Views
- 6 replies
- 1 kudos
Error writing data to Google Bigquery
Hi,I am able to read data from a Bigquery table ,But am getting error writing data to a table in BigqueryFollowed instuctions in this document.Connecting Databricks to BigQuery | Google Cloud%scalaimport scala.io.Sourceval contentCred = "/dbfs/FileSt...
- 7138 Views
- 6 replies
- 1 kudos
- 1 kudos
@udays22222 did you find any solution on this one? I face the same problem when I use Shared (Access mode) cluster. I can read but I cannot write with the error you mentioned.
- 1 kudos
- 3975 Views
- 14 replies
- 2 kudos
Autoloader with file notification mode sleeps for 5000ms multiple times
Using DBR 15.4, i'm ingesting streaming data from adls using autoloader with file notification mode enabled. This is an older code which is using foreachbatch sink to process the data before merging with tables in delta lake. IssueStreaming job, is u...
- 3975 Views
- 14 replies
- 2 kudos
- 2 kudos
@VZLA I just tested it and it seems this autoloader behaviour with available now trigger & file notification enabled, would remain the same with DLT pipeline, it sleeps 7 times each time sleeping for 5000ms before finally closing the stream, even tho...
- 2 kudos
- 2018 Views
- 3 replies
- 0 kudos
How To Parse a XML Column with string data type into multiple sql columns
Hi,I have a table with XML data in it which is stored in a column with STRING datatype. Can someone please help me on how to parse this XML into multiple sql columns.Below is the sample XML Table & desired output data Select * from default.SampleDat...
- 2018 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @LearnDB1234 ,Are you sure that this column stores xml as a string? To me it looks more like json string If so, you can use new VARIANT data type through parse_json function: %sql WITH src AS ( SELECT parse_json('{ "Status": { "Co...
- 0 kudos
- 5943 Views
- 1 replies
- 0 kudos
I keep on getting Parse_syntax_error on autoloader run foreachbatch
Hey guys, I keep on getting this error message when trying to call a function with soda DQ's: [PARSE_SYNTAX_ERROR] Syntax error at or near '{'. SQLSTATE: 42601 File <command-81221799516900>, line 4 1 dfBronze.writeStream \ 2 .foreachB...
- 5943 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sakuraDev , this looks like a Soda syntax issue. Try fixing the "fail" and "warn" fields in your Soda checks. For example, instead of writing: - missing_count(site) = 0: name: Ensure no null values fail: 1 warn: 0 Use Soda's thres...
- 0 kudos
- 4351 Views
- 1 replies
- 0 kudos
Looking for 75% coupon code for Data Engineering Associate Certification
Hi Everyone, I am Looking for 75% coupon code for Data Engineering Associate Certification . Can anyone Guide me how can get coupon code for certification.
- 4351 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Data_Engineer07 , Please reach out through https://www.databricks.com/company/contact regarding such requests. The corresponding team will guide on this request and let you know of its availability if there is.
- 0 kudos
- 2717 Views
- 2 replies
- 1 kudos
Databricks federation query why not support Oracle Database?
Hi,Based on the documents(https://docs.databricks.com/en/query-federation/index.html), Databricks federation query is not support Oracle for source. 1. Did you guys know the reason? (Is it depends on Oracle's speciality?)2. Is there another way to ru...
- 2717 Views
- 2 replies
- 1 kudos
- 1 kudos
@829023 There's limited support with respect to the pushdown and data types mapping as documented in our website: https://docs.databricks.com/en/query-federation/oracle.htmlThis was published recently, I believe in October, given your question was ra...
- 1 kudos
- 1167 Views
- 2 replies
- 0 kudos
Table Properties different for liquid clustering with Databricks version.
Dear all,Today, I tried the liquid clustering in Databricks, but after running it with two Databricks engine version, it showed different properties in the catalog explorer.1. Run with DBR version 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) it...
- 1167 Views
- 2 replies
- 0 kudos
- 0 kudos
Correct, like @holly rightly said this is just an updated way of representing the columns in a more structured or updated manner, it may also be matching a new value type. In both cases the table property is reflecting that LC was enabled. Our sugges...
- 0 kudos
- 2871 Views
- 3 replies
- 0 kudos
Passing Parameters in a Workflow pipeline
Hello Mates,I’m currently working on four workflows, all of which are connected to my own notebook.The four workflows should be started automatically one after the other and the only point that should be passed is the output of Workflow 1.The workflo...
- 2871 Views
- 3 replies
- 0 kudos
- 0 kudos
Yes the problem is that the .set and .get function works within the same job run, it does not pass to another job
- 0 kudos
- 3081 Views
- 6 replies
- 1 kudos
Logging to an external location via UC volume
The way I understand it, mount points are deprecated in UC. db.fs.mount() doesn't even seem to work in newer DB runtimes.But what is the solution when Databricks features don't allow using UC volumes? E.g. specifying a compute's logging path won't wo...
- 3081 Views
- 6 replies
- 1 kudos
- 1 kudos
As you cannot use volumes it seems that indeed this will be your only option
- 1 kudos
- 7843 Views
- 1 replies
- 0 kudos
Unable to login to Databricks Community Edition
Hello All,I have successfully created a databricks account and went to login to the community edition with the exact same login credentials as my account, but it tells me that the email/password are invalid. I can login with these same exact credenti...
- 7843 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Arihant!You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, feel free to submit a ticket with Databricks Support team for further assistance. Thank you.
- 0 kudos
- 2125 Views
- 2 replies
- 1 kudos
HOW TO SKIP A STEP FROM AN EXISTING WORKFLOW
Hi, can any one guide me in below scenario:Suppose we have created a workflow with 10 steps and our batches are running properly but due to some business requirement/testing purpose we have to skip step 4 and run the rest of job from step 5 onwords t...
- 2125 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @17abhishek ,You can try if/else conditions between the tasks with a Job parameter of "IsActiveTask" with a True or False.But it would be great if databricks team just add ability to simply disable some task from UI.
- 1 kudos
- 1425 Views
- 3 replies
- 0 kudos
DBR 16.0 spark read Azure Blob file Failed.
ala:570) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:354) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.NullPointerException at java.base/java.lang.Class.forName0(Native Method) at java.b...
- 1425 Views
- 3 replies
- 0 kudos
- 17339 Views
- 7 replies
- 1 kudos
FileNotFoundError: [Errno 2] No such file or directory: when I try to unzip .tar or .zip files it gives me this error
Hello, how are you? I have a small problem. I need to unzip some .zip, tar files. and gz inside these may have multiple files trying to unzip the .zip files i got this errorFileNotFoundError: [Errno 2] No such file or directory: but the files are in ...
- 17339 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @Alfredo Vallejos​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
