Databricks

quakenbush · 12-26-2023

Importing or cloning the .dbc Folder from "advanced-data-engineering-with-databricks" into my own workspace fails with a time-out. The folder is incomplete How can I fix this?I tried download and import the file and via URL...

quakenbush · 12-21-2023

HiWhat's the best way to deal with SCD2-styled tables in silver and/or gold layer while streaming.From what I've seen in the Professional Data Engineer videos, they usually go for SCD1 tables (simple updates or deletes)In a SCD2 scenario, we need to ...

quakenbush · 12-10-2023

Hi,I recently completed the fundamentals & advanced data engineer exam, yet I've got a question about Autoloader. Please don't go too hard on me, since I lack practical experience at this point in time Docs say this is incremental ingestion, so it's...

quakenbush · 01-30-2023

Since I am porting some code from Oracle to Databricks, I have another specific question.In Oracle there's something called Virtual Private Database, VPD. It's a simple security feature used to generate a WHERE-clause which the system will add to a u...

quakenbush · 01-18-2023

I am aware, I can load anything into a DataFrame using JDBC, that works well from Oracle sources. Is there an equivalent in Spark SQL, so I can combine datasets as well?Basically something like so - you get the idea...select lt.field1, rt.fie...

quakenbush · 12-29-2023

Thanks for your attention. After repeating like 5 times, it worked. I checked the import was complete by comparing with the Training material.I just passed the exam after review the Notebooks yesterday 

quakenbush · 12-22-2023

Maybe to_binary is what you're looking for...?https://docs.databricks.com/en/sql/language-manual/functions/to_binary.html

quakenbush · 12-22-2023

Just curious, could you point me to said QA session if it's a video or something? I'm not aware of such a limitation. You can use DLT's live streaming tables anywhere in the Medallion architecture, just make sure not to break stream composability by ...

quakenbush · 12-22-2023

I did some further reading and got the same conclusion. APPLY CHANGES might to the trick. However, I don't like the limitations. From Bronze to Silver I might need .foreachBatch to implement the JSON-logic and the attribute names (__start_at / __end_...

quakenbush · 12-22-2023

Well, one thing they emphasize in the 'Adavanced Data Engineer' Training is that job-clusters will terminate within 5 minutes after a job is completed. So this could be in support of your theory to lower costs. I think job-cluster are actually design...

Databricks

User Stats

User Activity

Time out importing DBC

Delta Lake, CFD & SCD2

Is Autoloader suitable to load full dumps?

Is there something like Oracle's VPD-Feature in Databricks?

Does Databricks offer something like Oracle's dblink?

Re: Time out importing DBC

Re: BINARY data type

Re: Change DLT table type from streaming to 'normal'

Re: Delta Lake, CFD & SCD2

Re: DLT Compute Resources - What Compute Is It???