- 475 Views
- 1 replies
- 0 kudos
how to generate log files on specific folders
Hello! My environments are like below, OS : Windows 11 Spark : spark-4.0.0-preview2-bin-hadoop3 And the configuration of spark files 'spark-defaults.conf' and 'log4j2.properties'spark-defaults.conf spark.eventLog.enabled true spark.event...
- 475 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aupres, Do you see any failures in spark logs? Few things to validate: It appears that the log files are not being generated in the specified directory due to a misconfiguration in your log4j2.properties fil Check the Appender Configuration: En...
- 0 kudos
- 8358 Views
- 3 replies
- 0 kudos
conversion of code from scala to python
does databricks community edition provides with databricks ML visualization for pyspark, just the same as provided in this link for scala. https://docs.azuredatabricks.net/_static/notebooks/decision-trees.html also please help me to convert this lin...
- 8358 Views
- 3 replies
- 0 kudos
- 0 kudos
you may explore the tool and services from Travinto Technologies . They have very good tools. We had explored their tool for our code coversion from Informatica, Datastage and abi initio to DATABRICKS , pyspark. Also we used for SQL queries, stored ...
- 0 kudos
- 9030 Views
- 3 replies
- 4 kudos
Converting SQL Code to SQL Databricks
I am new to Databricks. Please excuse my ignorance. My requirement is to convert the SQL query below into Databricks SQL. The query comes from EventLog table and the output of the query goes into EventSummaryThese queries can be found hereCREATE TABL...
- 9030 Views
- 3 replies
- 4 kudos
- 4 kudos
you may explore the tool and services from Travinto Technologies . They have very good tools. We had explored their tool for our code coversion from Informatica, Datastage and abi initio to DATABRICKS , pyspark. Also we used for SQL queries, stored ...
- 4 kudos
- 3537 Views
- 2 replies
- 0 kudos
Python UDF in Unity Catalog - spark.sql error
I'm trying to utilise the option to create UDFs in Unity Catalog. That would be a great way to have functions available in a fairly straightforward manner without e.g. putting the function definitions in an extra notebook that I %run to make them ava...
- 3537 Views
- 2 replies
- 0 kudos
- 0 kudos
I came across the same problem. inside unity catalog UDF creation, spark.sql or spark.table doesn't work.Adding from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() into the session doesn't work as wellDon't know how to sol...
- 0 kudos
- 26003 Views
- 9 replies
- 5 kudos
Resolved! Getting "Job aborted due to stage failure" SparkException when trying to download full result
I have generated a result using SQL. But whenever I try to download the full result (1 million rows), it is throwing SparkException. I can download the preview result but not the full result. Why ? What happens under the hood when I try to download ...
- 26003 Views
- 9 replies
- 5 kudos
- 5 kudos
Job aborted due to stage failure: Task 6506 in stage 46.0 failed 4 times, most recent failure: Lost task 6506.3 in stage 46.0 (TID 12896) (10.**.***.*** executor 12): java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (a...
- 5 kudos
- 5555 Views
- 6 replies
- 1 kudos
Error writing data to Google Bigquery
Hi,I am able to read data from a Bigquery table ,But am getting error writing data to a table in BigqueryFollowed instuctions in this document.Connecting Databricks to BigQuery | Google Cloud%scalaimport scala.io.Sourceval contentCred = "/dbfs/FileSt...
- 5555 Views
- 6 replies
- 1 kudos
- 1 kudos
@udays22222 did you find any solution on this one? I face the same problem when I use Shared (Access mode) cluster. I can read but I cannot write with the error you mentioned.
- 1 kudos
- 1698 Views
- 14 replies
- 2 kudos
Autoloader with file notification mode sleeps for 5000ms multiple times
Using DBR 15.4, i'm ingesting streaming data from adls using autoloader with file notification mode enabled. This is an older code which is using foreachbatch sink to process the data before merging with tables in delta lake. IssueStreaming job, is u...
- 1698 Views
- 14 replies
- 2 kudos
- 2 kudos
@VZLA I just tested it and it seems this autoloader behaviour with available now trigger & file notification enabled, would remain the same with DLT pipeline, it sleeps 7 times each time sleeping for 5000ms before finally closing the stream, even tho...
- 2 kudos
- 1042 Views
- 3 replies
- 0 kudos
How To Parse a XML Column with string data type into multiple sql columns
Hi,I have a table with XML data in it which is stored in a column with STRING datatype. Can someone please help me on how to parse this XML into multiple sql columns.Below is the sample XML Table & desired output data Select * from default.SampleDat...
- 1042 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @LearnDB1234 ,Are you sure that this column stores xml as a string? To me it looks more like json string If so, you can use new VARIANT data type through parse_json function: %sql WITH src AS ( SELECT parse_json('{ "Status": { "Co...
- 0 kudos
- 2262 Views
- 1 replies
- 0 kudos
I keep on getting Parse_syntax_error on autoloader run foreachbatch
Hey guys, I keep on getting this error message when trying to call a function with soda DQ's: [PARSE_SYNTAX_ERROR] Syntax error at or near '{'. SQLSTATE: 42601 File <command-81221799516900>, line 4 1 dfBronze.writeStream \ 2 .foreachB...
- 2262 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sakuraDev , this looks like a Soda syntax issue. Try fixing the "fail" and "warn" fields in your Soda checks. For example, instead of writing: - missing_count(site) = 0: name: Ensure no null values fail: 1 warn: 0 Use Soda's thres...
- 0 kudos
- 1648 Views
- 1 replies
- 0 kudos
Looking for 75% coupon code for Data Engineering Associate Certification
Hi Everyone, I am Looking for 75% coupon code for Data Engineering Associate Certification . Can anyone Guide me how can get coupon code for certification.
- 1648 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Data_Engineer07 , Please reach out through https://www.databricks.com/company/contact regarding such requests. The corresponding team will guide on this request and let you know of its availability if there is.
- 0 kudos
- 1902 Views
- 2 replies
- 1 kudos
Databricks federation query why not support Oracle Database?
Hi,Based on the documents(https://docs.databricks.com/en/query-federation/index.html), Databricks federation query is not support Oracle for source. 1. Did you guys know the reason? (Is it depends on Oracle's speciality?)2. Is there another way to ru...
- 1902 Views
- 2 replies
- 1 kudos
- 1 kudos
@829023 There's limited support with respect to the pushdown and data types mapping as documented in our website: https://docs.databricks.com/en/query-federation/oracle.htmlThis was published recently, I believe in October, given your question was ra...
- 1 kudos
- 530 Views
- 2 replies
- 0 kudos
Table Properties different for liquid clustering with Databricks version.
Dear all,Today, I tried the liquid clustering in Databricks, but after running it with two Databricks engine version, it showed different properties in the catalog explorer.1. Run with DBR version 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) it...
- 530 Views
- 2 replies
- 0 kudos
- 0 kudos
Correct, like @holly rightly said this is just an updated way of representing the columns in a more structured or updated manner, it may also be matching a new value type. In both cases the table property is reflecting that LC was enabled. Our sugges...
- 0 kudos
- 1200 Views
- 3 replies
- 0 kudos
Passing Parameters in a Workflow pipeline
Hello Mates,I’m currently working on four workflows, all of which are connected to my own notebook.The four workflows should be started automatically one after the other and the only point that should be passed is the output of Workflow 1.The workflo...
- 1200 Views
- 3 replies
- 0 kudos
- 0 kudos
Yes the problem is that the .set and .get function works within the same job run, it does not pass to another job
- 0 kudos
- 1214 Views
- 6 replies
- 1 kudos
Logging to an external location via UC volume
The way I understand it, mount points are deprecated in UC. db.fs.mount() doesn't even seem to work in newer DB runtimes.But what is the solution when Databricks features don't allow using UC volumes? E.g. specifying a compute's logging path won't wo...
- 1214 Views
- 6 replies
- 1 kudos
- 1 kudos
As you cannot use volumes it seems that indeed this will be your only option
- 1 kudos
- 7483 Views
- 1 replies
- 0 kudos
Unable to login to Databricks Community Edition
Hello All,I have successfully created a databricks account and went to login to the community edition with the exact same login credentials as my account, but it tells me that the email/password are invalid. I can login with these same exact credenti...
- 7483 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Arihant!You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, feel free to submit a ticket with Databricks Support team for further assistance. Thank you.
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 770 | |
| 349 | |
| 286 | |
| 253 |
