Databricks Community

turagittech

Hi All,I have set up a DLT pipleline for SQL Server to use CDC as per this instruction https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-pipeline I have it in principal working, however, it scans all databases a...

turagittech

Hi all,I need to do a batch load from sql server into Databricks. I have CC enabled on some tables. The simple appears to be union CDC and regular table to get a single set of records to load, but this appears to be fraught with risk of out of sequen...

turagittech · 04-06-2025

Hi all,I am working to resolve how to build tables in a development workspace catalog and then easily migrate the code to a production catalog without manually altering the schema name. For those unaware, you can't have the same catalog names in deve...

turagittech · 03-30-2025

Hi All,We are planning to store some mixed json files in blob store and read into Databricks. I am questioning whether we should have a container for each structure or if the various tools in Databricks can successfully read the different types. I ha...

turagittech · 03-11-2025

Hi,I have a blob storage area in Azure where json files are being created. I can create an external table on the storage blob container, but when new files are added I don't see extra rows to query the data. Is there a better approach to accessing th...

turagittech

I'll look at this once we go to production with the source files. I have split logs by file type to simplify this, but I'll go back and look again for the test space with mixed files

turagittech · 04-08-2025

I think you would want Delta Sharing. https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-sharing If you're Azure or GCP you need to find the equivalent.

turagittech · 04-08-2025

Thanks for this. Now to work out how much I want to work out alembic or dbt. I don't see any reason to go with Liquibase.Still an area for some improvements, actually a lot of improvement. Being able to build manageable, governed data warehouse schem...

turagittech · 04-06-2025

This doesn't hit the mark as I am referring to each json file representing a different table of data. I think multiple structures in a blob container confuse a lot of tools and that means you have to do file by file loading and that is going to be th...

turagittech · 04-03-2025

I would love to get an update on this. Git commands would be outstanding in some form. I have the same issue I have changed directory to the workspace. Ls shows the files in the repository, but git status fails.-rwxrwxrwx 1 root root 2386 Feb 13 01:1...

Databricks Community

User Stats

User Activity

DLT pipeline python stop scanning all databases in source

Batch reading from sql server tables with cdc on ssql server tables

Schema updating with CI/CD development in SQL

Reading different file structures for json files in blob stores

External Table refresh

Re: Reading different file structures for json files in blob stores

Re: Allow an external user to query SQL table in Databricks

Re: Schema updating with CI/CD development in SQL

Re: Reading different file structures for json files in blob stores

Re: How to use the git CLI in databricks?