Databricks Community

turagittech

Hi All,I have data in blobs which I am loading from blob store to Databricks delta tables. One of the blob types contains urls. From the Urls I want to extract knowledge from the path and query parts I can get those out easily with parse url. the pro...

turagittech · 08-21-2025

Hi All,I am struggling to understand how to manage credentials for azure storage across cluster when trying to use Azure python libraries within functions that may end up on the cluster worker nodes.I am building a task to load blobs from Azure stora...

turagittech · 08-14-2025

Hi all, when moving from development to testing a production one often needs to handle change values like the blob store or database server being differentI have seen that using widgets can be a useful way to have updateable values for Notebooks and ...

turagittech · 07-17-2025

Hi All,Sometimes to load data we want to only update a row based on changes in values. SCD1 type scenarios for a data warehouse. One approach is equivalency A=A, B=B etc. Another is generating a hash of all rows of interest, I believe pretty common. ...

turagittech · 05-25-2025

Hi All,I have set up a DLT pipleline for SQL Server to use CDC as per this instruction https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-pipeline I have it in principal working, however, it scans all databases a...

turagittech · 08-14-2025

Great, thanks. Speed in this case isn't critical as it's not processing massive amounts of data, well I hope not massive amounts at this time. It'll be some batch processes that can't use dlt.

turagittech · 08-14-2025

This works well when set up, If you're securely set up in Azure you will need to grant a privatelink to the underlying storage for their service to read data. For enhanced security I'd recommend your catalog for the other party then be in external st...

turagittech · 07-20-2025

I have been through this and spoke with our AM at Databricks. While in some analysis you'll see the VM costs exposed. They are hidden in the Azure Subscription costs screen as they are part of one of the higher order costs. Those being SQL Warehouse ...

turagittech · 07-17-2025

I thought I might follow up this after getting it all working with the help of my local Databricks office. AS the CDC has been crated it scans metadata for the server that you connect to. This may get altered in a future release, I have no idea as to...

turagittech · 05-15-2025

I'll look at this once we go to production with the source files. I have split logs by file type to simplify this, but I'll go back and look again for the test space with mixed files

Databricks Community

User Stats

User Activity

split parse_url output for the information

Credential Sharing Across Cluster Nodes - spark.conf()

Managing values that change between development and production

Concatenating a row to be able to hash

DLT pipeline python stop scanning all databases in source

Re: Managing values that change between development and production

Re: Data Transfer using Unity Catalog full implementation

Re: Understanding infra costs of Databricks compute

Re: DLT pipeline python stop scanning all databases in source

Re: Reading different file structures for json files in blob stores