cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Enzo_Bahrami
by • New Contributor III
  • 2947 Views
  • 2 replies
  • 0 kudos

Resolved! Input File Path from Autoloader in Delta Live Tables

Hello everyone!I was wondering if there is any way to get the subdirectories in which the file resides while loading while loading using Autoloader with DLT. For example:def customer(): return (  spark.readStream.format('cloudfiles')    .option('clou...

Data Engineering
Reply
  • 2947 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Parsa Bahraminejad​ We haven't heard from you since the last response from @Vigneshraja Palaniraj​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be...

  • 0 kudos
1 More Replies
CDICSteph
by • New Contributor
  • 2206 Views
  • 2 replies
  • 0 kudos

Need pattern for loading a million small XML files

Hi, looking for the right solution pattern for this scenario: We have millions of relatively small XML files (currently sitting in ADLS) that we have to load into delta lake. Each XML file has to be read, parsed, and pivoted before writing to a delta...

Data Engineering
Reply
  • 2206 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Steph Swierenga​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 0 kudos
1 More Replies
Anonymous
by • Not applicable
  • 2062 Views
  • 2 replies
  • 0 kudos

Safari errors in loading

Safari shows a white page, javascript console reports some errors: SyntaxError: Invalid regular expression: invalid group specifier nameUnhandled Promise Rejection: ChunkLoadError: Loading chunk 6956 failed.(missing: https://dd-databricks-live.cloud....

Data Engineering
Reply
  • 2062 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please make sure the safari is updated? Also, please try to clear the caches , history and restart the browser. Also, what are you trying to open and this error is surfaced?

  • 0 kudos
1 More Replies
BeginnerBob
by • New Contributor III
  • 2388 Views
  • 3 replies
  • 1 kudos

Loading Dimensions including SCDType2

I have a customer dimension and for every incremental load I am applying type2 or type1 to the dimension.This dimension is based off a silver table in my delta lake where I am applying a merge statement.What happens if I need to go back and track ad...

Data Engineering
Reply
  • 2388 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
BeginnerBob
New Contributor III
  • 1 kudos

Thanks werners,I was informed you could essentially recreate a type 2 dimensions from scratch, without reading the files 1 by 1, using the delta lake time shift. However, this doesn't seem to be the case and the only way to create this is to incremen...

  • 1 kudos
2 More Replies
aladda
by • Databricks Employee
  • 2331 Views
  • 1 replies
  • 0 kudos

Resolved! How can I speed up the loading of a large zipped CSV file in databricks

Data Engineering
Reply
  • 2331 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 0 kudos

gzip format is not splittable so the load process is sequential and thus slower. You can either try to split the CSV up into parts, gzip those separately and load them. Alternatively bzip is a splittable zip format that is better to work withOr you c...

  • 0 kudos
Labels