Databricks

thedatacrew · 01-16-2024

Hello,Could anyone please help regarding the scenario below?Scenario• I'm using the DLT SQL Language• Parquet files are landed each day from a source system.• Each day, the data contains the 7 previous days of data. The source system can have very la...

thedatacrew · 01-18-2024

Hi,I'm no expert in DLT, and I've only been using it for a few months. My reading of docs so far suggests streaming into bronze all your raw data using an autoloader and then using apply_changes into (SCD1/2) in silver.Finally, in the gold layer, use...

thedatacrew · 01-17-2024

Here is how my PyCharm/DataGrip is set up. You'll need a separate connection for each catalog.RegardsTobyhttps://thedatacrew.com

thedatacrew · 01-17-2024

You will also need to download and use the Databricks JDBC Driver https://www.databricks.com/spark/jdbc-drivers-download and install it in the User Drivers area of the connection properties.Your connection string jdbc:databricks://{SERVER_HOSTNAME}:4...

thedatacrew · 01-16-2024

I had the same problem in Datagrip, is there a startup script option in PyCharm database connection properties, if so try putting in use catalog your_unity_catalog_name;RegardsTobyhttps://thedatacrew.com

thedatacrew · 01-16-2024

I have found if you use a job cluster from a pool in ADF, it starts to create a cluster per data bricks adf activity, and you end up with more than 1 cluster running.I have a shared computer cluster for ADF with Photon/Unity enabled, and a fixed work...

Databricks

User Stats

User Activity

DTL - Delta Live Tables & Handling De-Duplication from Source Data

Re: DLT SCD type 2 in bronze, silver and gold? Is it possible?

Re: Introspecting catalogs and schemas JDBC in Pycharm

Re: Introspecting catalogs and schemas JDBC in Pycharm

Re: Introspecting catalogs and schemas JDBC in Pycharm

Re: Azure Data Factory and Photon