cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

yubin-apollo
by New Contributor II
  • 1249 Views
  • 4 replies
  • 0 kudos

COPY INTO skipRows FORMAT_OPTIONS does not work

Based on the COPY INTO documentation, it seems I can use `skipRows` to skip the first `n` rows. I am trying to load a CSV file where I need to skip a few first rows in the file. I have tried various combinations, e.g. setting header parameter on or ...

  • 1249 Views
  • 4 replies
  • 0 kudos
Latest Reply
karthik-kobai
New Contributor II
  • 0 kudos

@yubin-apollo: My bad - I had the skipRows in the COPY_OPTIONS and not in the FORMAT_OPTIONS. It works, please ignore my previous comment. Thanks

  • 0 kudos
3 More Replies
auser85
by New Contributor III
  • 2505 Views
  • 2 replies
  • 2 kudos

cannot convert Parquet type INT64 to Photon type double

I am trying to read in files via the COPY INTO command but I am getting this error lately for a certain subset of the data;`Error while reading file: Schema conversion error: cannot convert Parquet type INT64 to Photon type double`These are my option...

  • 2505 Views
  • 2 replies
  • 2 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 2 kudos

hey @Andrew Fogarty​ I also faced the same issue when I moved from the 7.3 LTS version to a higher runtime version so to mitigate this issue you can use the below cluster configuration spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsS...

  • 2 kudos
1 More Replies
dataexplorer
by New Contributor III
  • 4123 Views
  • 6 replies
  • 5 kudos

Resolved! COPY INTO generating duplicate rows in Delta table

Hello Everyone,I'm trying to bulk load tables from a SQL server database into ADLS as parquet files and then loading these files into Delta tables (raw/bronze). I had done a one off history/base load but my subsequent incremental loads (which had a d...

  • 4123 Views
  • 6 replies
  • 5 kudos
Latest Reply
dataexplorer
New Contributor III
  • 5 kudos

thanks for the guidance!

  • 5 kudos
5 More Replies
arul_parthiban
by New Contributor
  • 376 Views
  • 0 replies
  • 0 kudos

How to customize the result of COPY INTO command?

COPY INTO statement produces results in following format, which is more similar to INSERT INTO statement results; also it's a summary of all the files loaded. Is there a way to customize in way that it produces the detailed results at file level?

image
  • 376 Views
  • 0 replies
  • 0 kudos
Anonymous
by Not applicable
  • 1964 Views
  • 7 replies
  • 5 kudos

COPY INTO command can not recognise MAP type value from JSON file

I have a delta table in Databricks with single column of type map<string, string> and I have a data file in JSON format created by Hive 3 for the table with thecolumn of same type. And I want to load data from file to Databricks's table using COPY IN...

  • 1964 Views
  • 7 replies
  • 5 kudos
Latest Reply
jose_gonzalez
Moderator
  • 5 kudos

Hi Alexey,Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

  • 5 kudos
6 More Replies
Labels