Databricks Community

thedatacrew · 01-16-2024

Hello,Could anyone please help regarding the scenario below?Scenario• I'm using the DLT SQL Language• Parquet files are landed each day from a source system.• Each day, the data contains the 7 previous days of data. The source system can have very la...

thedatacrew

Hi,At the moment, my process is:-I am using ETL (Data Factory) to land paquet files in a raw landing zone. I keep all the source data here. So I can fully rebuild the data if I need to.i.e.source_system/schema/table/loadon_year=2024/loadon_month=08/l...

thedatacrew · 07-26-2024

Here is a freshly created Databrick NSG for VNET Injection on Azure. What new rules are required to be added? i.e. Source, Destination, Protocol?

thedatacrew · 01-18-2024

Hi,I'm no expert in DLT, and I've only been using it for a few months. My reading of docs so far suggests streaming into bronze all your raw data using an autoloader and then using apply_changes into (SCD1/2) in silver.Finally, in the gold layer, use...

thedatacrew · 01-17-2024

Here is how my PyCharm/DataGrip is set up. You'll need a separate connection for each catalog.RegardsTobyhttps://thedatacrew.com

thedatacrew · 01-17-2024

You will also need to download and use the Databricks JDBC Driver https://www.databricks.com/spark/jdbc-drivers-download and install it in the User Drivers area of the connection properties.Your connection string jdbc:databricks://{SERVER_HOSTNAME}:4...

Databricks Community

User Stats

User Activity

DTL - Delta Live Tables & Handling De-Duplication from Source Data

Re: DLT SCD type 2 in bronze, silver and gold? Is it possible?

Re: We are opening additional ports for Azure Databricks

Re: DLT SCD type 2 in bronze, silver and gold? Is it possible?

Re: Introspecting catalogs and schemas JDBC in Pycharm

Re: Introspecting catalogs and schemas JDBC in Pycharm