Databricks Community

noorbasha534

Hello allWe have a pipeline which uses auto-loader to load data from cloud object storage (ADLS) to a delta table. We use directory listing at the moment. And there exist around 20000 folders to be verified in ADLS every 30 mins to check for new data...

noorbasha534

Hello,Is there a sample code snippet that depicts end-to-end OIDC flow - imagine, there exists a service principal, interactive user who connect to an sql warehouse, get authenticated, and run some sql queries as part of a python script (jdbc/odbc) f...

noorbasha534

Hello all,I have a job that processes 50 tables - 25 belong to finance, 20 belong to master data, 5 belong to supply chain data domains.Now, imagine the job ran for 14 hours and did cost me 1000 euros on a day. If I like to attribute the per day cost...

noorbasha534

Hello all,Is it possible to persist Databricks job name into the Brooklyn audit tables data model when when a Databricks job calls DBT model?Currently, my colleagues persist audit information into fact & dimensional tables of the Brooklyn data model....

noorbasha534

Hi allIs it possible to configure the access_control_list block ((basically that contains permissions)) from a single place outside of Databricks job definition? This is changing as we are re-defining our permissions model, and each time resulting in...

noorbasha534

@szymon_dybczak ah sorry, let me rephrase. I tried the command initially on the delta table directly. That resulted the error. Then I tried on the check point. It did give me results though discovered on null for all the rows. Still, this does not s...

noorbasha534

@szymon_dybczak My target table is not a stream but regular delta table. I got this error - [STREAMING_TABLE_OPERATION_NOT_ALLOWED.NON_STREAMING_TABLE] The operation CLOUD_FILES_STATE is not allowed: `catalog_name`.`schema_name`.`table_name` is not a...

noorbasha534

@szymon_dybczak thanks for the reply. I have a single job that processes tables of multiple domains. I cannot split else the costs will blow up. The granular cost monitoring link shared talks about costs attribution for sql warehouses. We already loo...

noorbasha534

@Yogesh_378691 thanks. engineers need to remember to pass the job name as parameter, right? else, we have the risk of not seeing the job name in the audit tables. can you please elaborate little bit more on environment variables? is there a way to no...

noorbasha534

@WiliamRosa ah thanks much.Just a final question - it seems the code executes the SQL query & then captures the columns from the plan? If so, my need is to go through around 1000 sqls each day & get the columns used in filters/joins/group by of these...

Databricks Community

User Stats

User Activity

Figure out stale tables/folders being loaded by auto-loader

Any sample code snippet that contains end-to-end OIDC flow

Cost attribution based on table history statistics

Databricks job calling DBT - persist job name

Configure job access_control_list block from a single place outside of job definition

Re: Figure out stale tables/folders being loaded by auto-loader

Re: Figure out stale tables/folders being loaded by auto-loader

Re: Cost attribution based on table history statistics

Re: Databricks job calling DBT - persist job name

Re: EXPLAIN PLAN parser