cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Enzo_Bahrami
by New Contributor III
  • 1645 Views
  • 2 replies
  • 0 kudos

Resolved! Input File Path from Autoloader in Delta Live Tables

Hello everyone!I was wondering if there is any way to get the subdirectories in which the file resides while loading while loading using Autoloader with DLT. For example:def customer(): return (  spark.readStream.format('cloudfiles')    .option('clou...

  • 1645 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Parsa Bahraminejad​ We haven't heard from you since the last response from @Vigneshraja Palaniraj​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be...

  • 0 kudos
1 More Replies
CDICSteph
by New Contributor
  • 1397 Views
  • 2 replies
  • 0 kudos

Need pattern for loading a million small XML files

Hi, looking for the right solution pattern for this scenario: We have millions of relatively small XML files (currently sitting in ADLS) that we have to load into delta lake. Each XML file has to be read, parsed, and pivoted before writing to a delta...

  • 1397 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Steph Swierenga​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 0 kudos
1 More Replies
Anonymous
by Not applicable
  • 1373 Views
  • 2 replies
  • 0 kudos

Safari errors in loading

Safari shows a white page, javascript console reports some errors: SyntaxError: Invalid regular expression: invalid group specifier nameUnhandled Promise Rejection: ChunkLoadError: Loading chunk 6956 failed.(missing: https://dd-databricks-live.cloud....

  • 1373 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi, Could you please make sure the safari is updated? Also, please try to clear the caches , history and restart the browser. Also, what are you trying to open and this error is surfaced?

  • 0 kudos
1 More Replies
BeginnerBob
by New Contributor III
  • 1428 Views
  • 4 replies
  • 1 kudos

Loading Dimensions including SCDType2

I have a customer dimension and for every incremental load I am applying type2 or type1 to the dimension.This dimension is based off a silver table in my delta lake where I am applying a merge statement.What happens if I need to go back and track ad...

  • 1428 Views
  • 4 replies
  • 1 kudos
Latest Reply
Kaniz
Community Manager
  • 1 kudos

Hi @Lloyd Vickery​, We haven’t heard from you on the last response from @Werner Stinckens​, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to ot...

  • 1 kudos
3 More Replies
aladda
by Honored Contributor II
  • 1746 Views
  • 1 replies
  • 0 kudos
  • 1746 Views
  • 1 replies
  • 0 kudos
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

gzip format is not splittable so the load process is sequential and thus slower. You can either try to split the CSV up into parts, gzip those separately and load them. Alternatively bzip is a splittable zip format that is better to work withOr you c...

  • 0 kudos
Labels