Databricks Community

noorbasha534 · 09-29-2025

Dear allCan I create a single workload federation policy for all devops pipelines?Our set-up : we have code version controlled in Github repos. And, we use Azure DevOps pipelines to authenticate with Databricks via a service principal currently and d...

noorbasha534 · 09-11-2025

Hello allWe have a pipeline which uses auto-loader to load data from cloud object storage (ADLS) to a delta table. We use directory listing at the moment. And there exist around 20000 folders to be verified in ADLS every 30 mins to check for new data...

noorbasha534 · 09-04-2025

Hello,Is there a sample code snippet that depicts end-to-end OIDC flow - imagine, there exists a service principal, interactive user who connect to an sql warehouse, get authenticated, and run some sql queries as part of a python script (jdbc/odbc) f...

noorbasha534 · 09-04-2025

Hello all,I have a job that processes 50 tables - 25 belong to finance, 20 belong to master data, 5 belong to supply chain data domains.Now, imagine the job ran for 14 hours and did cost me 1000 euros on a day. If I like to attribute the per day cost...

noorbasha534 · 08-28-2025

Hello all,Is it possible to persist Databricks job name into the Brooklyn audit tables data model when when a Databricks job calls DBT model?Currently, my colleagues persist audit information into fact & dimensional tables of the Brooklyn data model....

noorbasha534 · 09-11-2025

@szymon_dybczak ah sorry, let me rephrase. I tried the command initially on the delta table directly. That resulted the error. Then I tried on the check point. It did give me results though discovered on null for all the rows. Still, this does not s...

noorbasha534 · 09-11-2025

@szymon_dybczak My target table is not a stream but regular delta table. I got this error - [STREAMING_TABLE_OPERATION_NOT_ALLOWED.NON_STREAMING_TABLE] The operation CLOUD_FILES_STATE is not allowed: `catalog_name`.`schema_name`.`table_name` is not a...

noorbasha534 · 09-04-2025

@szymon_dybczak thanks for the reply. I have a single job that processes tables of multiple domains. I cannot split else the costs will blow up. The granular cost monitoring link shared talks about costs attribution for sql warehouses. We already loo...

noorbasha534 · 09-02-2025

@Yogesh_Verma_ thanks. engineers need to remember to pass the job name as parameter, right? else, we have the risk of not seeing the job name in the audit tables. can you please elaborate little bit more on environment variables? is there a way to no...

noorbasha534 · 08-22-2025

@WiliamRosa ah thanks much.Just a final question - it seems the code executes the SQL query & then captures the columns from the plan? If so, my need is to go through around 1000 sqls each day & get the columns used in filters/joins/group by of these...

Databricks Community

User Stats

User Activity

Workload identity federation policy

Figure out stale tables/folders being loaded by auto-loader

Any sample code snippet that contains end-to-end OIDC flow

Cost attribution based on table history statistics

Databricks job calling DBT - persist job name

Re: Figure out stale tables/folders being loaded by auto-loader

Re: Figure out stale tables/folders being loaded by auto-loader

Re: Cost attribution based on table history statistics

Re: Databricks job calling DBT - persist job name

Re: EXPLAIN PLAN parser