- 2737 Views
- 2 replies
- 2 kudos
Resolved! Old files also getting added in dlt autoloader
So , I am using autoloader in a dlt pipeline for my data ingestion. I am using @Dlt.append_flow because I have data to load from multiple sources.When I load a new file say x it has 3 rows my target gets 3 rows. But next even if I don't load any file...
- 2737 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @ashraf1395, Just a few comments about your question: The cloudFiles source in Databricks is designed for incremental file processing. However, it depends on the checkpoint directory to track which files have been processed. The cloudFiles.include...
- 2 kudos
- 4755 Views
- 1 replies
- 0 kudos
foreach execution faulty with number of partitions >= worker cores
In order to download multiple wikipedia dumps, I collected the links in the list and wanted to use foreach method to iterate over those links and apply a UDF that downloads the data in the previously created volume structure. However, I ran into an i...
- 4755 Views
- 1 replies
- 0 kudos
- 0 kudos
It seems like the issue you're encountering with incomplete file downloads when using the foreach method and a UDF in Spark might be related to the number of partitions and how tasks are distributed across them. Here are a few points to consider: Ta...
- 0 kudos
- 5417 Views
- 1 replies
- 0 kudos
Can Databricks write query results to s3 in another account via the API
I work for a company where we are trying to create a Databrick's integration in node using the @DataBricks/sql package to query customers clusters or warehouses. I see documentation of being able to load data via a query from s3 using STS tokens whe...
- 5417 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you been able to get a response on this topic, based on the information I can see it might not be supported to write on an S3 outside your account
- 0 kudos
- 1322 Views
- 1 replies
- 0 kudos
Resolved! unvalidated the primary and foreign keys constraints?
Hello community,I'm inserting in a table defined (with primary key and foreign key set) some records in overwrite mode every moment I run a workflow where the task is defined. Why after inserting those records the DDL schema changes? Why I have my pr...
- 1322 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @jeremy98, When you use the "insert overwrite" mode in Databricks, it can lead to the schema being reset, which includes the removal of primary and foreign key constraints. This happens because the "insert overwrite" operation essentially replaces...
- 0 kudos
- 13541 Views
- 1 replies
- 0 kudos
Concurrent issue on delta lake insert update
Hey team! I need your help on delta lake let me explain the scenario of mine.Scenario: ive a table in delta lake and ive 2 databricks workflows running parallely which has insert and update tasks to do.My delta table is partitioned with country codeM...
- 13541 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, @Dhanushn In response to your question, the community contains the following information:https://community.databricks.com/t5/community-platform-discussions/concurrent-update-to-delta-throws-error/td-p/65599https://kb.databricks.com/en_US/delta/in...
- 0 kudos
- 1841 Views
- 1 replies
- 0 kudos
how to generate log files on specific folders
Hello! My environments are like below, OS : Windows 11 Spark : spark-4.0.0-preview2-bin-hadoop3 And the configuration of spark files 'spark-defaults.conf' and 'log4j2.properties'spark-defaults.conf spark.eventLog.enabled true spark.event...
- 1841 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @aupres, Do you see any failures in spark logs? Few things to validate: It appears that the log files are not being generated in the specified directory due to a misconfiguration in your log4j2.properties fil Check the Appender Configuration: En...
- 0 kudos
- 9896 Views
- 3 replies
- 0 kudos
conversion of code from scala to python
does databricks community edition provides with databricks ML visualization for pyspark, just the same as provided in this link for scala. https://docs.azuredatabricks.net/_static/notebooks/decision-trees.html also please help me to convert this lin...
- 9896 Views
- 3 replies
- 0 kudos
- 0 kudos
you may explore the tool and services from Travinto Technologies . They have very good tools. We had explored their tool for our code coversion from Informatica, Datastage and abi initio to DATABRICKS , pyspark. Also we used for SQL queries, stored ...
- 0 kudos
- 12734 Views
- 3 replies
- 4 kudos
Converting SQL Code to SQL Databricks
I am new to Databricks. Please excuse my ignorance. My requirement is to convert the SQL query below into Databricks SQL. The query comes from EventLog table and the output of the query goes into EventSummaryThese queries can be found hereCREATE TABL...
- 12734 Views
- 3 replies
- 4 kudos
- 4 kudos
you may explore the tool and services from Travinto Technologies . They have very good tools. We had explored their tool for our code coversion from Informatica, Datastage and abi initio to DATABRICKS , pyspark. Also we used for SQL queries, stored ...
- 4 kudos
- 6745 Views
- 2 replies
- 0 kudos
Python UDF in Unity Catalog - spark.sql error
I'm trying to utilise the option to create UDFs in Unity Catalog. That would be a great way to have functions available in a fairly straightforward manner without e.g. putting the function definitions in an extra notebook that I %run to make them ava...
- 6745 Views
- 2 replies
- 0 kudos
- 0 kudos
I came across the same problem. inside unity catalog UDF creation, spark.sql or spark.table doesn't work.Adding from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() into the session doesn't work as wellDon't know how to sol...
- 0 kudos
- 31921 Views
- 9 replies
- 5 kudos
Resolved! Getting "Job aborted due to stage failure" SparkException when trying to download full result
I have generated a result using SQL. But whenever I try to download the full result (1 million rows), it is throwing SparkException. I can download the preview result but not the full result. Why ? What happens under the hood when I try to download ...
- 31921 Views
- 9 replies
- 5 kudos
- 5 kudos
Job aborted due to stage failure: Task 6506 in stage 46.0 failed 4 times, most recent failure: Lost task 6506.3 in stage 46.0 (TID 12896) (10.**.***.*** executor 12): java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (a...
- 5 kudos
- 7908 Views
- 6 replies
- 1 kudos
Error writing data to Google Bigquery
Hi,I am able to read data from a Bigquery table ,But am getting error writing data to a table in BigqueryFollowed instuctions in this document.Connecting Databricks to BigQuery | Google Cloud%scalaimport scala.io.Sourceval contentCred = "/dbfs/FileSt...
- 7908 Views
- 6 replies
- 1 kudos
- 1 kudos
@udays22222 did you find any solution on this one? I face the same problem when I use Shared (Access mode) cluster. I can read but I cannot write with the error you mentioned.
- 1 kudos
- 5142 Views
- 14 replies
- 2 kudos
Autoloader with file notification mode sleeps for 5000ms multiple times
Using DBR 15.4, i'm ingesting streaming data from adls using autoloader with file notification mode enabled. This is an older code which is using foreachbatch sink to process the data before merging with tables in delta lake. IssueStreaming job, is u...
- 5142 Views
- 14 replies
- 2 kudos
- 2 kudos
@VZLA I just tested it and it seems this autoloader behaviour with available now trigger & file notification enabled, would remain the same with DLT pipeline, it sleeps 7 times each time sleeping for 5000ms before finally closing the stream, even tho...
- 2 kudos
- 2409 Views
- 3 replies
- 0 kudos
How To Parse a XML Column with string data type into multiple sql columns
Hi,I have a table with XML data in it which is stored in a column with STRING datatype. Can someone please help me on how to parse this XML into multiple sql columns.Below is the sample XML Table & desired output data Select * from default.SampleDat...
- 2409 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @LearnDB1234 ,Are you sure that this column stores xml as a string? To me it looks more like json string If so, you can use new VARIANT data type through parse_json function: %sql WITH src AS ( SELECT parse_json('{ "Status": { "Co...
- 0 kudos
- 6471 Views
- 1 replies
- 0 kudos
I keep on getting Parse_syntax_error on autoloader run foreachbatch
Hey guys, I keep on getting this error message when trying to call a function with soda DQ's: [PARSE_SYNTAX_ERROR] Syntax error at or near '{'. SQLSTATE: 42601 File <command-81221799516900>, line 4 1 dfBronze.writeStream \ 2 .foreachB...
- 6471 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sakuraDev , this looks like a Soda syntax issue. Try fixing the "fail" and "warn" fields in your Soda checks. For example, instead of writing: - missing_count(site) = 0: name: Ensure no null values fail: 1 warn: 0 Use Soda's thres...
- 0 kudos
- 4598 Views
- 1 replies
- 0 kudos
Looking for 75% coupon code for Data Engineering Associate Certification
Hi Everyone, I am Looking for 75% coupon code for Data Engineering Associate Certification . Can anyone Guide me how can get coupon code for certification.
- 4598 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Data_Engineer07 , Please reach out through https://www.databricks.com/company/contact regarding such requests. The corresponding team will guide on this request and let you know of its availability if there is.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
