cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

fuselessmatt
by Contributor
  • 2272 Views
  • 3 replies
  • 0 kudos

Omitting columns in an INSERT statement does not seem to work despite meeting the requirements

We want to use the INSERT INTO command with specific columns as specified in the official documentation. The only requirements for this are​️ Databricks SQL warehouse version 2022.35 or higher️ Databricks Runtime 11.2 and above​and the behaviour shou...

  • 2272 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Fusselmanwog​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 0 kudos
2 More Replies
cmilligan
by Contributor II
  • 1247 Views
  • 3 replies
  • 0 kudos

Undescriptive error when trying to insert overwrite into a table

I have a query that I'm trying to insert overwrite into a table. In an effort to try and speed up the query I added a range join hint. After adding it I started getting the error below.I can get around this though by creating a temporary view of the ...

Screenshot_20230118_104626
  • 1247 Views
  • 3 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Could you share your code and the full error stack trace please? Check the driver logs for the full stack trace.

  • 0 kudos
2 More Replies
elgeo
by Valued Contributor II
  • 1003 Views
  • 1 replies
  • 1 kudos

Merge didn't fail while inserting wrong data type values

Hello. During some example cases we were running, in order to identify how Databricks treats possible wrong actions we could make, we noticed that merge doesn't fail while inserting different data type values from the ones in the corresponding table....

Target table schema merge_failure image merge_failure3
  • 1003 Views
  • 1 replies
  • 1 kudos
Latest Reply
elgeo
Valued Contributor II
  • 1 kudos

Hello. Any update on this please? Thank you in advance

  • 1 kudos
BorislavBlagoev
by Valued Contributor III
  • 2687 Views
  • 9 replies
  • 3 kudos

Resolved! Tring to create incremental pipeline but fails when I try to use outputMode "update"

def upsertToDelta(microBatchOutputDF, batchId): microBatchOutputDF.createOrReplaceTempView("updates")   microBatchOutputDF._jdf.sparkSession().sql(""" MERGE INTO old o USING updates u ON u.id = o.id WHEN MATCHED THEN UPDATE SE...

  • 2687 Views
  • 9 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Delta table/file version is too old. Please try to upgrade it as described here https://docs.microsoft.com/en-us/azure/databricks/delta/versioning​

  • 3 kudos
8 More Replies
RobertWalsh
by New Contributor II
  • 15001 Views
  • 11 replies
  • 0 kudos

Dataframe Write Append to Parquet Table - Partition Issue

Hello, I am attempting to append new json files into an existing parquet table defined in Databricks. Using a dataset defined by this command (dataframe initially added to a temp table): val output = sql("select headers.event_name, to_date(from_unix...

0693f000007OoJYAA0 0693f000007OoJZAA0
  • 15001 Views
  • 11 replies
  • 0 kudos
Latest Reply
anil_s_langote
New Contributor II
  • 0 kudos

We came across similar situation we are using spark 1.6.1, we have a daily load process to pull data from oracle and write as parquet files, this works fine for 18 days of data (till 18th run), the problem comes after 19th run where the data frame l...

  • 0 kudos
10 More Replies
Labels